MOTIVATION: Neurodegenerative disorders influence millions of people worldwide, and uncovering the pathogenesis is of urgent need. Many efforts have been made to detect or predict neurodegenerative disorders, while exploring the pathogenesis has been ignored from a systemic perspective. RESULTS: To handle this issue, we propose a novel and powerful method, referred to as Pathogenesis-aware Mutual-Assistance Classification and Regression Optimization (Pa-MACRO). First, Pa-MACRO incorporates a mutual-assistance bidirectional mapping technique with a joint-embedding fine-grained interpretability module. This can extract the intrinsic factors and their interactions of multimodal pathogenesis. Second, our method can simultaneously classify an at-risk individual and predict the severity triggered by neurodegenerative disorders. Furthermore, to address the small sample size issue and the high-dimensional issue, we meticulously incorporate a semi-supervised cooperative learning method to integrate unlabeled data and extend it to a chromosome-wide setting in the spirit of divide-and-conquer. The Alzheimer's Disease Neuroimaging Initiative (ADNI) database was used to evaluate Pa-MACRO. Without bells and whistles, Pa-MACRO establishes new state-of-the-art results in various settings while maintaining superior interpretability, verifying its power and versatility in revealing the pathogenesis of neurodegenerative disorders. AVAILABILITY AND IMPLEMENTATION: The software is publicly available at https://github.com/ZJ-Techie/Pa-MACRO.

Segment Anything Model (SAM) has demonstrated state-of-the-art performance in most segmentation tasks. However, due to insufficient training in the medical domain, SAM’s ability to generalize to medical images is limited. Although preliminary efforts have fine-tuned SAM for the medical domain, the fine-tuned model still struggles with variability in medical tasks. Some recent studies have explored weakly supervised learning to mitigate SAM’s performance degradation in the medical domain. However, the effectiveness of weakly supervised learning is heavily dependent on the quality of weakly supervised information, with performance significantly dropping as the quality declines. Doctors’ attention is closely related to the target area during diagnosis. Integrating gaze information into SAM’s adaptation process for medical image segmentation enhances efficiency and significantly improves performance in medical tasks. In this paper, we first propose a Gaze-assisted medical segment Anything Model (GAM), which utilizes gaze information to enable the adaptation of SAM in medical images following doctor’s attention. It has two innovations: 1) Feature-level adaptation: Gaze Alignment (GA) learning makes the feature-level adaptation follow the doctor’s attention which mines the human guidance from gaze heatmaps and guides model to extract general features for downstream tasks. 2) Output-level adaptation: Gaze-Balance (GB) learning makes the output-level adaptation follow the doctor’s attention which utilizes gaze heatmaps to enhance the human-focused area and solve the problem of over/under segmentation from the output-level. Our promising results on 7 tasks with 12 targets have demonstrated the powerful adaptation ability of our GAM in the medical domain. Our GAM demonstrates significant potential for low-cost clinical assistance in medical diagnosis, enabling SAM to adapt to the medical image domain without disrupting clinical workflows. We have released the full source code on https://github.com/Ruiz1026/GAM.

Survival prediction is crucial for cancer patients as it provides essential early prognostic information for treatment planning and decision making. Despite impressive performance , current multi-modal survival prediction methods that integrate pathology and genomic data face two main challenges: (1) Whole-slide images (WSIs) generally exhibit hierarchical structures, but the interactions of phenotypes at different resolutions remain unexplored. More importantly, the potential semantic discrepancy arising from diverse resolutions is often ignored. (2) The absence of effective interactions between the inherent hierarchical structures of WSIs and genomic data. To address these challenges, in this paper, we propose Cross-modal Dual-interaction Transformer (CD-Former), a robust hierarchical framework for multi-modal survival prediction. Our CD-Former involves two key components: (1) an Multimodal Cross-Scale Calibration (MCSC) module for effectively capturing correlations across multiple resolutions and calibrating fine-grained features, thereby bridging the semantic discrepancy caused by different WSI resolutions; and (2) a hierarchical interaction module termed Multi-modal Dual-interaction (M2Di) for fully exploring multi-resolution cross-modal correlations and interactions, which comprises a Patch-level Cross-Attention Block (PCAB) and a Region-level Cross-Attention Block (RCAB) to investigate cross-modal associations between patch- or region-level features of WSI and genomic data. Additionally, we employ a scale-oriented WSI enhancer to capture the interactions among various components of WSIs. The experimental results demonstrate the effectiveness of our proposed framework, which achieves state-of-the-art performance compared to previous studies.

Functional brain connectivity networks capture complex relationships and temporal evolution between brain regions, which have become increasingly important for diagnosing neurological disorders. However, existing methods, which are primarily based on vector or graph representations, struggle to adequately characterize the intricate spatio-temporal topological architecture of functional brain networks. Additionally, they predominantly rely on data-driven paradigms and lack priors pertaining to cross-windows network interactions. To address these issues, we propose a spatio-temporal hypergraph attention network framework for brain network analysis. Specifically, we first propose a temporal attention network architecture embedded with temporal similarity-driven prior knowledge, which effectively extracts long-range dependency information from fMRI by combining multi-head self-attention mechanisms and cross-window temporal prior knowledge. Second, we design a hierarchical hypergraph generation module that fuses local and global brain topological information to achieve multi-scale modeling of high-order spatio-temporal structures. Additionally, the spatial attention network, developed based on transformer architecture, employs hypergraph message passing mechanisms to effectively construct multi-level spatial interaction relationships between brain regions. Finally, a multi-layer perceptron (MLP) is adopted for classification. Experiments on the ADNI and PD datasets demonstrate that our method outperforms several state-of-the-art approaches in diagnostic performance and provides discriminative graph features for relevant brain disease diagnosis.

Segmenting the brain magnetic resonance (MR) images to region-of-interest (ROI) is a fundamental step for many medical image analysis tasks. Convolutional neural networks (CNNs) excel in learning the high-level contextual features for image segmentation. However, such high-level features are low-order features, which cannot reflect the complex appearance patterns of brain MR images. Intuitively, using the high-order features can enhance the performance of CNNs. Therefore, in this paper, we propose a novel Efficient Covariance Network (EfficientCovNet) that models pairwise voxel dependency features and applies it to the brain ROI segmentation tasks. Our EfficientCovNet consists of two pathways: a pairwise voxel dependency feature learning pathway that uses a novel covariance convolution to efficiently capture the pairwise features from MR images, and a contextual feature learning pathway that extracts high-level contextual features using convolutional operations. The pairwise features and contextual features are then fused together to boost brain ROI segmentation performance. Experimental results on five datasets, i.e., IXI, LONI-LPBA40, OASIS, ADNI, and CC359 datasets, demonstrate that our EfficientCovNet achieves superior performance for brain ROI segmentation in comparison with the state-of-the-art methods.

Tissue segmentation in pathological images plays a crucial role for the diagnosis and prognosis of human cancers. However, due to the complexity of tumor micro-environment, it is difficult to annotate all tissue types especially for the categories with small tissue proportions, which limits the ability of the traditional tissue segmentation models to these tissue types with zero training samples. To address the above issues, we present a novel architecture, ZSPMLG, that relies on pathology vision-language foundation model (i.e., CONCH) to learn pixel-wise classifiers for both seen and unseen tissue types based on their text descriptions. Specifically, we firstly apply large language model (LLM) to generate the descriptions for both seen and unseen tissue categories, followed by feeding them to the CONCH text encoder to acquire their corresponding prototypes that are shared by both vision and semantic space. By considering that the textual descriptions of specific tissue categories can be observed from the pathological images at different scales of magnification, our ZSPMLG consists of Mixture of Local Experts (MoLE) and Mixture of Global Experts (MoGE) modules, where MoLE performs the specialized decoding that can map individual scale patch-level representation to dense pixel-level representation, while MoGE aims at fusing the multi-scale representations together. Finally, a convolutional layer is designed to map the pixel-level representation to the category prototype for tissue segmentation on both seen and unseen categories. We evaluate our method on three datasets and the experimental results demonstrate the superiority of our method on both seen and unseen tissue categories.

Source-free domain adaptation (SFDA) adapts a pre-trained model from a labeled source domain to an unlabeled target domain without source data access, preserving privacy. While SFDA is common in computer vision, it remains largely unexplored in time series analysis, where existing methods struggle to capture temporal dynamics and often produce overconfident predictions on out-of-distribution samples. We propose MAsk And imPUte (MAPU), which tackles temporal consistency through a novel imputation task, where randomly masked time series signals are recovered within the learned embedding space. During adaptation, a dedicated temporal imputer guides the target model to generate features that maintain temporal consistency with source features. However, MAPU relies on standard softmax predictions, leading to overconfident predictions on target samples that fall outside the source domain's support. To address this limitation, we introduce Evidential-MAPU (E-MAPU), which leverages evidential uncertainty estimation to identify these out-of-support samples and adapts the feature extractor to map them closer to the source domain's support, while maintaining the classifier fixed. Extensive experiments on five real-world time series datasets demonstrate significant performance improvements over existing methods. Our approaches effectively handle various time series domain adaptation challenges while maintaining computational efficiency, achieving state-of-the-art performance through its uncertainty-aware adaptation strategy.

The recent advance of deep learning has shown great potential for nucleus detection which plays an important role in the histopathological examination. However, such accurate and reliable deep learning models usually need enough labeled data for training, which makes active learning an appealing learning paradigm to reduce the annotation efforts by experts. In open-set environments, active learning encounters the challenge that the unlabeled data usually contain non-target samples from the unknown classes, resulting in the failure of most active learning methods. Although active learning has been explored in many open-set classification tasks, research on active learning for nucleus detection in the open-set environment remains unexplored. To address the above issues, we propose a two-stage active learning framework designed for nucleus detection in the open-set environment (i.e., OpAL4ND). In the first stage, we propose a prototype-based query strategy based on the auxiliary detector to select a candidate set from known classes as pure as possible. In the second stage, we further query the most uncertain and representative samples from the candidate set for the nucleus detection task relying on the target detector. We evaluate the performance of our method on two nucleus detection datasets (i.e., the NuCLS and PanNuke datasets), and the experimental results indicate that our method can not only improve the selection quality on the known classes, but also achieve higher detection accuracy with lower annotation burden in comparison with the existing studies. Code is available at https://github.com/onbut/OpAL4ND.

Multi-modal ultrasound combines tissue information from multiple imaging perspectives, enabling more comprehensive lesion assessment. However, conventional multi-view learning methods typically assume uniform modality quality, ignoring variability caused by imaging noise and patient-specific factors. This oversight limits diagnostic reliability, especially when some modalities provide uncertain or conflicting information. To address this, we identify two key challenges in multi-modal ultrasound fusion: 1) how to quantify modality-wise uncertainty, and 2) how to resolve conflicts among predictions. We propose a novel method, termed TMUF (Trustworthy Multi-modal Ultrasound Fusion), which dynamically integrates information from different modalities through uncertainty calibration and conflict resolution. Specifically, we introduce a cross-modal uncertainty calibration regularizer to estimate evidence-based uncertainty across modalities, aligning uncertainty with prediction correctness. We further develop a credibility-aware fusion strategy that evaluates cross-modal consistency and uncertainty to distinguish credible from non-credible modalities, assigning fusion weights accordingly. We validate TMUF on public and private datasets for breast lesion and liver cancer diagnosis. The proposed method achieves diagnostic accuracies of 88.00% and 92.08%, respectively, outperforming state-of-the-art baselines. These results demonstrate the effectiveness of TMUF in enhancing diagnostic accuracy and robustness for multi-modal ultrasound.

The characteristic of neurodegenerative disorders is the progressive impairment of memory and other cognitive functions. However, these existing imaging genetic methods only use longitudinal imaging phenotypes straightforwardly, ignoring the latent pattern of the longitudinal data in the progression process. The phenotypes across multiple time-points may exhibit the latent pattern that can be used to facilitate the understanding of the progression process. Accordingly, in this paper, we explore underlying complementary information from multiple time-points and simultaneously seek the underlying latent representation. With the complementarity of multiple time-points, the latent representation depicts data more comprehensively than each individual time-point, therefore mining effective longitudinal phenotype latent pattern representation. Specifically, we first propose two latent pattern representation (LPR) for longitudinal imaging phenotypes: linear LPR (lLPR), based on linear relationships between latent representation and each time-point, and nonlinear LPR (nonlLPR), based on neural networks to deal with nonlinear relationships. Then, we calculate the imaging genetic association based on the latent pattern representation. Finally, we conduct the experiments on both synthetic and real longitudinal imaging genetic data. Related experimental results validate that our proposed approach outperforms several competing algorithms, establishes strong associations, and discovers consistent longitudinal imaging genetic biomarkers, thereby guiding disease interpretation.

Multiomics co-learning is a powerful analytical paradigm that has benefited biomedical studies substantially. However, due to the diverse information and complex relationships of multiomics data, naive multi-view learning methods usually run into spurious correlations and biased signatures irrelevant to the diseases of interest. Therefore, the learned representations and cross-omics associations cannot translate into clinical knowledge for disease prediction. This issue becomes particularly severe when clinical data are limited and scarce. To handle this issue, we propose a novel and powerful scheme, referred to as the Causality-driven Trustworthy Multi-View maPping approach (Cad-TMVP). Specifically, we design a fined multi-directional mapping module to extract co-expression patterns across different modalities and capture fine-grained interpretability factors. We also meticulously design dynamic mechanisms to facilitate adaptive loss-term reweighting and trustworthy integration of multiple modalities. Cad-TMVP enhances downstream tasks by developing a cooperative learning module that simultaneously performs automated diagnosis and result interpretation. Furthermore, we develop an efficient search strategy and support computation to reduce the high computational burden, making our approach practicable. We conduct extensive experiments on different types of multiomics data. The proposed method establishes new state-of-the-art results in various settings while maintaining excellent interpretability. Thus, it sets a potentially newparadigm in trustworthy multi-modal learning and verifies its flexibility and versatility in real biomedical applications.

Real-time tissue tracking is a fundamental task in liver ultrasound applications. Due to the periodic nature of liver motion, historical trajectories can offer valuable priors for target localization, particularly when foreground-background distinction is weak. However, existing trackers often exploit these trajectories as shortcuts, relying excessively on periodic respiratory patterns rather than true object appearance matching. In this work, we revisit liver tracking from a causal perspective and propose CUSTrack, a method that mitigates periodicity bias by decomposing and correcting the total causal effect of historical trajectories. We define periodicity bias as the direct causal effect of past states and eliminate it via counterfactual reasoning, preserving 'good' trajectory priors while suppressing 'bad' periodic bias. To ensure identifiability, we incorporate a deconfounding module that removes latent confounders from fused feature representations. Extensive experiments on liver ultrasound datasets demonstrate that CUSTrack achieves superior tracking accuracy and robustness under challenging conditions.

Multiplex immunohistochemistry (mIHC) images have the potential to assess the complex tumor microenvironment by simultaneously detecting multiple markers within a single tissue section, however, the acquisition of mIHC images in clinical labs is both time-consuming and costly. Hence, applying machine learning-based virtual staining techniques for rapid generation of different mIHC markers has become a considerable alternative. The existing bio-image based virtual staining models generate the distributions of different markers independently, which have limited interpretability and overlook the fact that the exploration of potential interrelationships among these markers can help determine the localization of each individual marker. To address the above issues, we propose an explainable prototypical multi-task generation framework (i.e., ProtoMTG) to simultaneously generate multiple mIHC markers. Specifically, ProtoMTG involves a multi-task prototype layer that can capture the relationship among different virtual staining tasks by learning the shared and task-specific prototypes. Then, in the proto-attention layer, both task-specific and shared prototypes will be re-weighted and combined to instruct the generation of different mIHC markers. In ProtoMTG, we also design the novel prototypical activation and diversity losses to learn better prototype representation for the virtual staining task. To evaluate the performance of our method, we develop three benchmark mIHC datasets on different organs (i.e., colon, liver and stomach). The experimental results indicate that our method can not only outperform the existing image generation models, but also have good explainable ability for the virtual staining of mIHC markers. The code and dataset are available at: https://jj-zhou-code.github.io/ProtoMTG-website/.

The domain gap resulting from mismatches in acquisition details like protocol and scanner between training and test data hinders the deployment of the trained model in clinical practice. To address this issue, Continual test-time adaptation (CTTA) has been proposed to adapt the source model to continually changing unlabeled domains without accessing the source data. Existing methods learn an image-level visual prompt for target domains and inject the trainable prompt into the input space. However, they either combine the input with a prompt of equal scale or determine the prompt injection position through complex strategies such as uncertainty estimation or Fourier Transform. These approaches substantially increase the number of trainable parameters and computational burden, especially in high-dimensional medical imaging data. To overcome these challenges, we propose the Efficient Deformable Convolutional Prompt (EDCP), which leverages the inductive bias of convolution to reduce trainable parameters compared to standard prompts. We further enhance convolution by making it deformable, addressing fine-grained domain shifts at the pixel level through an offset branch. To improve training efficiency and balance parameters between the convolution and offset branches, we decompose the offset transformation into two parts, storing one in an offset bank that also serves as a domain indicator. This bank accelerates training by skipping test images similar to those already stored. Prompt updates are guided by layer-wise alignment of source-target statistics without unfreezing batch normalization layers. Extensive experiments demonstrate the superiority of our method in 2D and 3D medical image segmentation tasks.

RNA modifications are essential regulators of gene expression and cellular function, with over 170 distinct types identified to date. Among them, N6-methyladenosine is the most prevalent internal modification in mammalian messenger RNA. Nanopore direct RNA sequencing offers the potential to study RNA modifications at high resolution, but challenges persist for single-molecule resolution detection due to signal noise, low modification frequency, and read-level variability. To address these challenges, we propose MambaNano, a dualbranch deep learning framework designed for direct modeling of raw current signals in m6A detection. The model incorporates a temporal branch to capture long-range signal dependencies and a spatial branch to encode statistical and positional descriptors. These representations are fused through a Mambadriven integration module, followed by an attention-based pooling mechanism that enhances per-site signal summarization. We evaluate MambaNano on nanopore sequencing data from human cell lines. Experimental results demonstrate that the model achieves state-of-the-art detection of$\mathbf{m 6 A}$modifications at the single-molecule level. Visualization further demonstrates that the model responds differently to hypermethylated and hypomethylated modified sequence motifs, confirming its capacity to extract biologically meaningful patterns. Our results highlight the potential of structured state-space models combined with multi-branch processing for accurate and generalizable RNA modification analysis. The source code of MambaNano is available at https://anonymous.4open.science/r/MambaNano-EB62.

Decoding visual stimuli from human brain activity is a fundamental challenge in cognitive neuroscience and neuroimaging. While recent advances in deep learning have significantly improved the performance of fMRI-to-image decoding, most existing methods overlook the issue of inter-subject variability in fMRI data, which leads to poor generalization across subjects. Current approaches often rely on partially shared model architectures that offer limited generalization and still require subject-specific components, restricting their applicability to unseen subjects. To address this limitation, we propose BrainX, a universal brain decoding framework that constructs a unified fMRI encoder and image generator to achieve subject-agnostic modeling. Specifically, we introduce a feature disentanglement mechanism that extracts subject-shared features from the fMRI embeddings, which are then fed into the image generator to reconstruct visual stimuli. This design eliminates the need for subject-specific models and significantly enhances cross-subject generalization. Additionally, we develop a neuro-geometric fMRI representation learning method that projects 3D cortical structures onto a 2D surface space, effectively mitigating the inaccuracies caused by imprecise geodesic distance estimation in 3D Euclidean space. Extensive experiments on the Natural Scenes Dataset (NSD) demonstrate that BrainX consistently outperforms existing state-of-the-art methods across three decoding settings: within-subject, cross-subject with finetuning, and cross-subject without finetuning.

The rapid development of spatial transcriptomics (ST) allows researchers to measure the spatial-level gene expression in tissues. Although powerful, the cost for collecting the ST data is expensive, and thus several studies aim to predict gene expression in ST by utilizing their corresponding H/E stained pathology images. The existing ST based gene expression prediction models either adopt the pre-trained networks or rely on the handcrafted features to describe the pathology images, which still lack a systematic way to combine them together to define a spot-level representation that can reflect the topological profiles of different spots. On the other hand, all the ST based gene prediction models treat the prediction task for each gene independently, which overlook the fact that the exploration of potential interrelationships among them can help improve the prediction performance for individual genes. To address the above issues, we propose a multi-modal topology-embedded graph learning algorithm guided by prior Gene Ontology similarity information (i.e., M2TGLGO) to predict the spatial resolved genes from pathology images. Specifically, M2TGLGO co-learns the image representation of different spots from both deep and handcrafted features by considering the within-modal and inter-modal interactions. Next, to keep the topological structure among different spots, a spatial-oriented ranking module is also incorporated to preserve their neighborhood similarity information. Finally, we present a Gene Ontology knowledge guided graph neural network for simultaneously predicting multiple gene expressions by considering their functional associations. We evaluate our method on three public available ST datasets, the experimental results show the effectiveness of our M2TGLGO in comparison with the existing studies.

Vision-language models (VLMs) like CLIP (Contrastive Language-Image Pre-Training) have seen remarkable success in visual recognition, highlighting the increasing need to safeguard the intellectual property (IP) of well-trained models. Effective IP protection extends beyond ensuring authorized usage; it also necessitates restricting model deployment to authorized data domains, particularly when the model is fine-tuned for specific target domains. However, current IP protection methods often rely solely on the visual backbone, which may lack sufficient semantic richness. To bridge this gap, we introduce IP-CLIP, a lightweight IP protection strategy tailored to CLIP, employing a prompt-based learning approach. By leveraging the frozen visual backbone of CLIP, we extract both image style and content information, incorporating them into the learning of IP prompt. This strategy acts as a robust barrier, effectively preventing the unauthorized transfer of features from authorized domains to unauthorized ones. Additionally, we propose a style-enhancement branch that constructs feature banks for both authorized and unauthorized domains. This branch integrates self-enhanced and cross-domain features, further strengthening IP-CLIP’s capability to block features from unauthorized domains. Finally, we present new three metrics designed to better balance the performance degradation of authorized and unauthorized domains. Comprehensive experiments in various scenarios demonstrate its promising potential for application in IP protection tasks for VLMs.

The integrative analysis of histopathological images and genomic data has received increasing attention for survival prediction of human cancers. However, the existing studies always hold the assumption that full modalities are available. As a matter of fact, the cost for collecting genomic data is high, which sometimes makes genomic data unavailable in testing samples. A common way of tackling such incompleteness is to generate the genomic representations from the pathology images. Nevertheless, such strategy still faces the following two challenges: (1) The gigapixel whole slide images (WSIs) are huge and thus hard for representation. (2) It is difficult to generate the genomic embeddings with diverse function categories in a unified generative framework. To address the above challenges, we propose a Conditional Latent Differentiation Variational AutoEncoder (LD-CVAE) for robust multimodal survival prediction, even with missing genomic data. Specifically, a Variational Information Bottleneck Transformer (VIBTrans) module is proposed to learn compressed pathological representations from the gigapixel WSIs. To generate different functional genomic features, we develop a novel Latent Differentiation Variational AutoEncoder (LD-VAE) to learn the genomic and function-specific posteriors for the genomic embeddings with diverse functions. Finally, we use the product-of-experts technique to integrate the genomic posterior and image posterior for the joint latent distribution estimation in LD-CVAE. We test the effectiveness of our method on five different cancer datasets, and the experimental results demonstrate its superiority in both complete and missing modality scenarios. The code is released†.

The prediction of nanoparticles (NPs) distribution is crucial for the diagnosis and treatment of tumors. Recent studies indicate that the heterogeneity of tumor microenvironment (TME) highly affects the distribution of NPs across tumors. Hence, it has become a research hotspot to generate the NPs distribution by the aid of multi-modal TME components. However, the distribution divergence among multi-modal TME components may cause side effects i.e., the best unimodal model may outperform the joint generative model. To address the above issues, we propose a Divergence-Aware Multi-Modal Diffusion model (i.e., DAMM-Diffusion) to adaptively generate the prediction results from uni-modal and multi-modal branches in a unified network. In detail, the uni-modal branch is composed of the U-Net architecture while the multi-modal branch extends it by introducing two novel fusion modules i.e., Multi-Modal Fusion Module (MMFM) and Uncertainty-Aware Fusion Module (UAFM). Specifically, the MMFM is proposed to fuse features from multiple modalities, while the UAFM module is introduced to learn the uncertainty map for cross-attention computation. Following the individual prediction results from each branch, the Divergence-Aware Multi-Modal Predictor (DAMMP) module is proposed to assess the consistency of multi-modal data with the uncertainty map, which determines whether the final prediction results come from multi-modal or uni-modal predictions. We predict the NPs distribution given the TME components of tumor vessels and cell nuclei, and the experimental results show that DAMM-Diffusion can generate the distribution of NPs with higher accuracy than the comparing methods. Additional results on the multi-modal brain image synthesis task further validate the effectiveness of the proposed method. The code is released†.

Multimodal fusion provides cross-modality information to understand the human brain from different perspectives that may be missed in single modality analysis. Supervised fusion focuses on extracting multimodal patterns related to specific clinical measures by further incorporating a prior interested reference. However, existing supervised fusion methods cannot extract component that have weak correlations with the reference, which may be lost during the optimization process. Here, we propose a reference-guided parallel independent component analysis (RG-PICA) aiming at identifying multimodal covarying features related to interested reference through global optimization. The intra-modality independence, the inter-modality correlation, and the correlation between modalities and the reference are maximized globally. Simulations show that RG-PICA can accurately extract multimodal features correlated with the weak related reference while keeping cross-modality linkage comparing with seven fusion methods. In real data application, RG-PICA reveals co-varying patterns in schizophrenia (SZ) that links with cognition and correlates between modalities. These results demonstrate RG-PICA can jointly optimize for target components that correlate with the reference while keeping cross-modality linkage. This approach can improve the meaningful detection of reliable reference-linked multimodal brain patterns for brain disorders.

Dynamic functional connectivity (DFC) can capture the neural activity changes over time in the brain. Most existing DFC constructions rely on sliding windows, which can be highly impacted by window type and width. In addition, previous methods fail to fully optimize for discriminative spatial-temporal (ST) information and can suffer from inter-site heterogeneity, resulting in suboptimal sensitivity to brain disorders. Here, we propose a novel DFC model by combining ST attention-based bidirectional long short-term memory (BiLSTM) and multi-source domain adaptation (DA) to extract inherent ST information and reduce inter-site heterogeneity. An adaptive similarity sparse representation (SR)-based Kalman filter is proposed to obtain DFC with accurate connectivity strength at each time point. ST attention modules are integrated into BiLSTM to capture discriminative ST features with a maximum mean discrepancy (MMD)-constrained module for multi-source DA. Experimental results show that our method achieves high accuracy (90.67%±2.43%) in discriminating schizophrenia (SZ) from controls, outperforming 7 DA, 6 ST, and 5 DFC models. These results demonstrate the effectiveness of the proposed DFC model, which can be used to investigate multi-site fMRI DFC for the diagnosis of brain disorders.

Domain adaptation (DA) for cardiac ultrasound image segmentation is clinically significant and valuable. However, previous domain adaptation methods are prone to be affected by the incomplete pseudo label and low-quality target to source images. Human-centric domain adaptation has great advantages of human cognitive guidance to help model adapt to target domain and reduce reliance on labels. Doctor gaze trajectories contains a large amount of cross-domain human guidance. To leverage gaze information and human cognition for guiding domain adaptation, we propose gaze-assisted human-centric domain adaptation (GAHCDA), which reliably guides the domain adaptation of cardiac ultrasound images. GAHCDA includes following modules: (1) Gaze Augment Alignment (GAA): GAA enables the model to obtain human cognition general features to recognize segmentation target in different domain of cardiac ultrasound images like humans. (2) Gaze Balance Loss (GBL): GBL fused gaze heatmap with outputs which makes the segmentation result structurally closer to the target domain. The experimental results show that our proposed framework is able to segment cardiac ultrasound images more effectively in the target domain than GAN-based methods and other self-train based methods and shown great potential in clinical application.

As a highly interconnected complex network system, the brain exhibits changes in interactions due to common brain disorders. Studying changes in brain network interactions can help us quantitatively analyze functional network patterns and changes in these patterns that are linked to brain disorders. However, relationships between brain regions estimated by most current approaches use a single connectivity that does not fully reflect multiple interactions. Here, we propose a novel functional connectivity (FC) construction method, which can estimate both cooperative and competitive (C-C) relationships between the same regions of interest (ROIs) through improved Ising model. We redefine the Ising dynamic equation to represent pairwise interactions from single to C-C relationships. Results show that the estimated C-C connectivities are normally distributed, with intra-subjects’ (n=970) similarity being consistently and significantly higher than inter-subjects’ similarity across datasets. C-C FCs between occipital, parietal, temporal cortex and the limbic system of schizophrenia (SZ, n=178) are more competitive, while healthy control (HC, n=219) tends to be more cooperative. Group differences in C-C patterns between SZ and HC show significant differences in frontal, parietal and occipital regions. The proposed C-C approach provide new insights into the brain dysfunction in SZ, which can also be applied to investigate other brain disorders.

Source-Free Domain Adaptation (SFDA) aims to adapt a pre-trained model from an annotated source domain to an unlabelled target domain without accessing the source data, thereby preserving data privacy. While existing SFDA methods have proven effective in reducing reliance on source data, they struggle to perform well on multivariate time series (MTS) due to their failure to consider the intrinsic spatial correlations inherent in MTS data. These spatial correlations are crucial for accurately representing MTS data and preserving invariant information across domains. To address this challenge, we propose Temporal Restoration and Spatial Rewiring (TERSE), a novel and concise SFDA method tailored for MTS data. Specifically, TERSE comprises a customized spatial-temporal feature encoder designed to capture the underlying spatial-temporal characteristics, coupled with both temporal restoration and spatial rewiring tasks to reinstate latent representations of the temporally masked time series and the spatially masked correlated structures. During the target adaptation phase, the target encoder is guided to produce spatially and temporally consistent features with the source domain by leveraging the source pre-trained temporal restoration and spatial rewiring networks. Therefore, TERSE can effectively model and transfer spatial-temporal dependencies across domains, facilitating implicit feature alignment. In addition, as the first approach to simultaneously consider spatial-temporal consistency in MTS-SFDA, TERSE can also be integrated as a versatile plug-and-play module into established SFDA methods. Extensive experiments on three real-world time series datasets demonstrate the effectiveness and versatility of our approach. Our code is available at https://github.com/Tokenmw/TERSE-master.

Test-time adaptation aims to adapt pre-trained deep neural networks using solely online unlabelled test data during inference. Although TTA has shown promise in visual applications, its potential in time series contexts remains largely unexplored. Existing TTA methods, originally designed for visual tasks, may not effectively handle the complex temporal dynamics of real-world time series data, resulting in suboptimal adaptation performance. To address this gap, we propose Augmented Contrastive Clustering with Uncertainty-aware Prototyping (ACCUP), a straightforward yet effective TTA method for time series data. Initially, our approach employs augmentation ensemble on the time series data to capture diverse temporal information and variations, incorporating uncertainty-aware prototypes to distill essential characteristics. Additionally, we introduce an entropy comparison scheme to selectively acquire more confident predictions, enhancing the reliability of pseudo labels. Furthermore, we utilize augmented contrastive clustering to enhance feature discriminability and mitigate error accumulation from noisy pseudo labels, promoting cohesive clustering within the same class while facilitating clear separation between different classes. Extensive experiments conducted on three real-world time series datasets demonstrate the effectiveness and generalization potential of the proposed method, advancing the underexplored realm of TTA for time series data. Our code is available at https://github.com/Tokenmw/ACCUP-main.

Dynamic functional brain networks (DFBNs) are powerful tools in neuroscience research. Recent studies reveal that DFBNs contain heterogeneous neural nodes with more extensive connections and more drastic temporal changes, which play pivotal roles in coordinating the reorganization of the brain. Moreover, the spatio-temporal patterns of these nodes are modulated by the brain's historical states. However, existing methods not only ignore the spatio-temporal heterogeneity of neural nodes, but also fail to effectively encode the temporal propagation mechanism of heterogeneous activities. These limitations hinder the deep exploration of spatio-temporal relationships within DFBNs, preventing the capture of abnormal neural heterogeneity caused by brain diseases. To address these challenges, this paper propose a neuro-heterogeneity guided temporal graph learning strategy (NeuroH-TGL). Specifically, we first develop a spatio-temporal pattern decoupling module to disentangle DFBNs into topological consistency networks and temporal trend networks that align with the brain's operational mechanisms. Then, we introduce a heterogeneity mining module to identify pivotal heterogeneity nodes that drive brain reorganization from the two decoupled networks. Finally, we design temporal propagation graph convolution to simulate the influence of the historical states of heterogeneity nodes on the current topology, thereby flexibly extracting heterogeneous spatio-temporal information from the brain. Experiments show that our method surpasses several state-of-the-art methods, and can identify abnormal heterogeneous nodes caused by brain diseases.

Decoding natural language text from non-invasive brain signals, such as functional magnetic resonance imaging (fMRI), remains a central challenge in brain-computer interface research. While recent advances in large language models (LLMs) have enabled open-vocabulary fMRI-to-text decoding, existing frameworks typically process the entire fMRI sequence in a single step, leading to performance degradation when handling long input sequences due to memory overload and semantic drift. To address this limitation, we propose a brain-inspired sequential fMRI-to-text decoding framework that mimics the human cognitive strategy of segmented and inductive language processing. Specifically, we divide long fMRI time series into consecutive segments aligned with optimal language comprehension length. Each segment is decoded incrementally, followed by a wrap-up mechanism that summarizes the semantic content and incorporates it as prior knowledge into subsequent decoding steps. This sequence-wise approach alleviates memory burden and ensures semantic continuity across segments. In addition, we introduce a text-guided masking strategy integrated with a masked autoencoder (MAE) framework for fMRI representation learning. This method leverages attention distributions over key semantic tokens to selectively mask the corresponding fMRI time points, and employs MAE to guide the model toward focusing on neural activity at semantically salient moments, thereby enhancing the capability of fMRI embeddings to represent textual information. Experimental results on the two datasets demonstrate that our method significantly outperforms state-of-the-art approaches, with performance gains increasing as decoding length grows.

The whole-slide pathology images (WSIs) are widely recognized as the golden standard for cancer survival analysis. However, due to the high-resolution of WSIs, the existing studies require dividing WSIs into patches and identify key components before building the survival prediction system, which is time-consuming and cannot reflect the overall spatial organization of WSIs. Inspired by the fact that the spatial interactions among different tumor microenvironment (TME) components in WSIs are associated with the cancer prognosis, some studies attempt to capture the complex interactions among different TME components to improve survival predictions. However, they require extra efforts for building the TME segmentation model, which involves substantial annotation workloads on different TME components and is independent to the construction of the survival prediction model. To address the above issues, we propose ZTSurv, a novel end-to-end cancer survival analysis framework via efficient zero-shot TME segmentation on low-resolution WSIs. Specifically, by leveraging tumor infiltrating lymphocyte (TIL) maps on the 50x down-sampled WSIs, ZTSurv enables zero-shot segmentation on other two important TME components (i.e., tumor and stroma) that can reduce the annotation efforts from the pathologists. Then, based on the visual and semantic information extracted from different TME components, we construct a heterogeneous graph to capture their spatial intersections for clinical outcome prediction. We validate ZTSurv across four cancer cohorts derived from The Cancer Genome Atlas (TCGA), and the experimental results indicate that our method can not only achieve superior prediction results but also significantly reduce the computational costs in comparison with the state-of-the-art methods.

Prompt learning has emerged as a promising paradigm for adapting pre-trained vision-language models (VLMs) to few-shot whole slide image (WSI) classification by aligning visual features with textual representations, thereby reducing annotation cost and enhancing model generalization. Nevertheless, existing methods typically rely on slide-level prompts and fail to capture the subtype-specific phenotypic variations of histological entities (e.g., nuclei, glands) that are critical for cancer diagnosis. To address this gap, we propose Multi-scale Attribute-enhanced Prompt Learning (MAPLE), a hierarchical framework for few-shot WSI classification that jointly integrates multi-scale visual semantics and performs prediction at both the entity and slide levels. Specifically, we first leverage large language models (LLMs) to generate entity-level prompts that can help identify multi-scale histological entities and their phenotypic attributes, as well as slide-level prompts to capture global visual descriptions. Then, an entity-guided cross-attention module is proposed to generate entity-level features, followed by aligning with their corresponding subtype-specific attributes for fine-grained entity-level prediction. To enrich entity representations, we further develop a cross-scale entity graph learning module that can update these representations by capturing their semantic correlations within and across scales. The refined representations are then aggregated into a slide-level representation and aligned with the corresponding prompts for slide-level prediction. Finally, we combine both entity-level and slide-level outputs to produce the final prediction results. Results on three cancer cohorts confirm the effectiveness of our approach in addressing few-shot pathology diagnosis tasks.

Integrating and analyzing multiple omics datasets, such as genomics, environmental influences, and imaging endophenotypes, has yielded an abundance of candidate biomarkers. However, translating such findings into beneficial clinical knowledge for disease prediction remains challenging. This becomes even more challenging when studying interpretable high-order feature interactions such as gene-environment interaction (G$\times $E) to understand the etiology. To fill this gap, we draw on the idea of mutual-assistance (MA) learning and accordingly propose a fresh and powerful scheme, referred to as mutual-assistance causal biomarker discovery and stable disease prediction approach (MA-CBxDP). Specifically, we design an interpretable bi-directional mapping framework, integrated with a causal feature interaction module, to extract co-expression patterns across different modalities and identify trustworthy biomarkers including G$\times $E. A cooperative prediction module is further incorporated to ensure accurate diagnosis and identification of causal effects for pathogenesis. Importantly, biomarker discovery and disease prediction can mutually reinforce each other, helping to provide novel insights into chronic diseases. Furthermore, in light of the large computational burden incurred by the high-dimensional interactions, we devise a rapid strategy and extend it to a more practical but challenging chromosome-wide setting. We conduct extensive experiments on two databases under three tasks, i.e. multimodal correlation, disease diagnosis, and trait prediction. MA-CBxDP establishes new state-of-the-art results in predicting clinical scores and disease status classification, while maintaining exceptional interpretability, verifying its flexibility and versatility in practical applications.

MOTIVATION: Discovering genetic variations underpinning brain disorders is important to understand their pathogenesis. Indirect associations or spurious causal relationships pose a threat to the reliability of biomarker discovery for brain disorders, potentially misleading or incurring bias in subsequent decision-making. Unfortunately, the stringent selection of reliable biomarker candidates for brain disorders remains a predominantly unexplored challenge. RESULTS: In this article, to fill this gap, we propose a fresh and powerful scheme, referred to as the Causality-aware Genotype intermediate Phenotype Correlation Approach (Ca-GPCA). Specifically, we design a bidirectional association learning framework, integrated with a parallel causal variable decorrelation module and sparse variable regularizer module, to identify trustworthy causal biomarkers. A disease diagnosis module is further incorporated to ensure accurate diagnosis and identification of causal effects for pathogenesis. Additionally, considering the large computational burden incurred by high-dimensional genotype-phenotype covariances, we develop a fast and efficient strategy to reduce the runtime and prompt practical availability and applicability. Extensive experimental results on four simulation data and real neuroimaging genetic data clearly show that Ca-GPCA outperforms state-of-the-art methods with excellent built-in interpretability. This can provide novel and reliable insights into the underlying pathogenic mechanisms of brain disorders. AVAILABILITY AND IMPLEMENTATION: The software is publicly available at https://github.com/ZJ-Techie/Ca-GPCA.

MOTIVATION: Oxford Nanopore Technologies (ONT) direct RNA sequencing (dRNA-seq) offers high-resolution, single-molecule analysis but is hindered by the lack of robust multiplex barcoding methods. Existing approaches struggle to accurately demultiplex raw nanopore signals, failing to capture both local patterns and long-range dependencies. This limitation underscores the requirement for advanced solutions to improve accuracy, efficiency, and adaptability in sequencing workflows. We present DemuxTrans, a hybrid deep learning framework that integrates Multi-Layer Feature Fusion, Transformers, and Temporal Convolutional Networks (TCN) for precise barcode demultiplexing. RESULTS: DemuxTrans achieves state-of-the-art performance across multiple datasets by effectively balancing local feature extraction, global context modeling, and long-term dependency capture, excelling in metrics such as accuracy, recall and F1-score. These results demonstrate DemuxTrans as a scalable, efficient solution for barcode demultiplexing in nanopore sequencing, enabling precise identification of multiplexed RNA samples and improving throughput in transcriptomic and epigenomic analyses. AVAILABILITY AND IMPLEMENTATION: The code and datasets are publicly available on https://github.com/LiyuanShu116/Demuxtrans.

Deep learning has been extensively applied in medical image reconstruction, where Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) represent the predominant paradigms, each possessing distinct advantages and inherent limitations: CNNs exhibit linear complexity with local sensitivity, whereas ViTs demonstrate quadratic complexity with global sensitivity. The emerging Mamba has shown superiority in learning visual representation, which combines the advantages of linear scalability and global sensitivity. In this study, we introduce MambaMIR, an Arbitrary-Masked Mamba-based model with wavelet decomposition for joint medical image reconstruction and uncertainty estimation. A novel Arbitrary Scan Masking (ASM) mechanism "masks out" redundant information to introduce randomness for further uncertainty estimation. Compared to the commonly used Monte Carlo (MC) dropout, our proposed MC-ASM provides an uncertainty map without the need for hyperparameter tuning and mitigates the performance drop typically observed when applying dropout to low-level tasks. For further texture preservation and better perceptual quality, we employ the wavelet transformation into MambaMIR and explore its variant based on the Generative Adversarial Network, namely MambaMIR-GAN. Comprehensive experiments have been conducted for multiple representative medical image reconstruction tasks, demonstrating that the proposed MambaMIR and MambaMIR-GAN outperform other baseline and state-of-the-art methods in different reconstruction tasks, where MambaMIR achieves the best reconstruction fidelity and MambaMIR-GAN has the best perceptual quality. In addition, our MC-ASM provides uncertainty maps as an additional tool for clinicians, while mitigating the typical performance drop caused by the commonly used dropout.

EEG-based emotion recognition is crucial for understanding human affective states, offering valuable insights into diverse fields like mental health monitoring and humancomputer interaction. Recent advancements in graph learning have significantly impacted EEG emotion recognition due to their ability to model the complex, dynamic relationships within brain networks. However, current methods often neglect the interplay between shared and individual correlations among EEG channels. Furthermore, individual variations in EEG patterns lead to distributional shifts that hinder the generalization of existing approaches. This paper proposes a novel Two-stage domain Adaptation with Hybrid Adaptive Graph learning (TAHAG) for EEG emotion recognition. TAHAG first employs hybrid adaptive graph learning to capture both shared and individual spatial characteristics of the EEG signals, dynamically integrating their contributions. Feature attention mechanisms are then incorporated to refine node features and enhance the model's discriminability. To address distributional variations, TAHAG utilizes a two-stage domain adaptation strategy. This strategy involves aligning the refined node features across different domains through discrepancy alignment. Subsequently, adversarial training captures domain-invariant summarized features of the entire graph. Extensive experiments on three public datasets demonstrate the superiority of TAHAG compared to existing methods. Furthermore, visualization of neuronal activity reveals significant brain regions and inter-channel relationships relevant to EEG emotion recognition.

Multi-modal emotion recognition has attracted much attention in human-computer interaction, because it provides complementary information for the recognition model. However, the distribution drift among subjects and the heterogeneity of different modalities pose challenges to multi-modal emotion recognition, thereby limiting its practical application. Most of the current multi-modal emotion recognition methods are difficult to suppress above uncertainties in fusion. In this paper, we propose a cross-subject multi-modal emotion recognition framework, which jointly learns subject-independent representation and common feature between EEG and eye movements. First, we design the dynamic adversarial domain adaptation for cross-subject distribution alignment, dynamically selecting source domains in training. Second, we simultaneously capture intra-modal and inter-modal emotion-related features by both self-attention and cross-attention mechanisms, thus obtaining the robust and complementary representation of emotional information. Then, two contrastive loss functions are imposed on above network to further reduce inter-modal heterogeneity, and mine higher-order semantic similarity between synchronously collected multi-modal data. Finally, we used the output of the softmax layer as the predicted value. The experimental results on several multi-modal emotion datasets with EEG and eye movements demonstrate that our method is significantly superior to the state-of-the-art emotion recognition approaches.

In the brain imaging genomic tasks, it is challenging to provide accurate prior knowledge for estimating the association between quantitative traits (QTs) extracted from neuroimaging and genetic markers like single-nucleotide polymorphisms (SNPs). The hidden structural patterns in data limit the discovery of disease-related biomarkers. To this end, we present a deep ring-wise block network (RB-Net) for association analysis and brain disease diagnosis. Specifically, we first construct a new hidden structural pattern, namely, ring-wise block pattern, that satisfies both block and ring properties within the data before the association analysis. Subsequently, a RB-Net is developed via using an auto-encoder (AE) to represent imaging genomic data. Furthermore, we design the approach for joint association learning and automated brain disease diagnosis. Additionally, the optimization scheme based on alternating update is presented for solve the built ring-wise block-perception layer model. The performance of the designed method has been experimentally assessed on the brain imaging genomic data from the Alzheimer's Disease Neuroimaging Initiative (ADNI). The results validate that the proposed approach outperforms some competing approaches, establishes strong associations, and identifies crucial regions of interest (ROIs) across different imaging phenotypes associated with genetic risk biomarkers, thereby guiding disease interpretation and diagnosis prediction.

Electroencephalography (EEG) biometrics draws increasing attention in high-security requirements due to its advantages of anti-spoofing, live traits, and non-duplicated. However, existing EEG datasets, which rely on external stimuli or task-specific instructions for data collection, often intertwine identity-related information with biases such as emotional states, cognitive tasks, and disease markers. Besides, EEG signals are time-varying, while identity information within EEG signals is relatively fixed, which poses challenges for extracting identity features from EEG to perform accurate person identification. This high correlation hampers the promotion of brainprint recognition in real-life applications. In this paper, we propose a disentangled representation learning based identity recognition framework, which disentangles the EEG signal into intrinsic identity-related information and biased identity-invariant information, thus enhancing the performance of EEG biometrics. First, two parallel encoders are used to extract intrinsic identity-relevant and bias identity-irrelevant factors, respectively, and each encoder consists of a temporal filter module and a novel spatial-temporal attention module. Then, we further refine the disentanglement process through a correlation-driven loss that minimizes factor similarity across spatial-temporal and global representational domains. Adversarial training and reconstruction regularization are introduced to facilitate the identity and biased representations to be independent and complementary to each other. Additionally, we extend supervised contrastive learning to the component level, minimizing cross-component similarity and encouraging each component to independently reflect its unique information, thereby improving the disentanglement efficacy. Our proposed framework achieves state-of-the-art performance on diverse datasets encompassing emotional, motor imagery, and pathological conditions, demonstrating the robustness and effectiveness of our proposed brainprint identity recognition model.

Dynamic functional brain network (DFBN) can flexibly describe the time-varying topological connectivity patterns of the brain, and show great potential in brain disease diagnosis. However, most of the existing DFBN analysis methods focus on capturing the dynamic interaction at the brain region level, ignoring the spatio-temporal topological evolution across time windows. Moreover, they are difficult to suppress interfering connections in DFBNs, which leads to a diminished capacity for discerning the intrinsic structures that are intimately linked to brain disorders. To address these issues, we propose a topological evolution graph learning model to capture disease-related spatio-temporal topological features in DFBNs. Specifically, we first take the hubness of adjacent DFBN as the source domain and the target domain in turn, and then use Wasserstein distance (WD) and Gromov-Wasserstein distance (GWD) to capture the brain's evolution law at the node and edge levels, respectively. Furthermore, we introduce the principle of relevant information to guide the topology evolution graph to learn the structures that are most relevant to brain diseases yet least redundant information between adjacent DFBNs. On this basis, we develop a high-order spatio-temporal model with multi-hop graph convolution to collaboratively extract long-range spatial and temporal dependencies from the topological evolution graph. Extensive experiments show that the proposed method outperforms the current state-of-the-art methods, and can effectively reveal the information evolution mechanism between brain regions across windows.

Multimodal fusion provides multiple benefits over single modality analysis by leveraging both shared and complementary information from different modalities. Notably, supervised fusion enjoys extensive interest for capturing multimodal co-varying patterns associated with clinical measures. A key challenge of brain data analysis is how to handle confounds, which, if unaddressed, can lead to an unrealistic description of the relationship between the brain and clinical measures. Current approaches often rely on linear regression to remove covariate effects prior to fusion, which may lead to information loss, rather than pursue the more global strategy of optimizing both fusion and covariates removal simultaneously. Thus, we propose "CR-mCCAR" to jointly optimize for confounds within a guided fusion model, capturing co-varying multimodal patterns associated with a specific clinical domain while also discounting covariate effects. Simulations show that CR-mCCAR separate the reference and covariate factors accurately. Functional and structural neuroimaging data fusion reveals co-varying patterns in attention deficit/hyperactivity disorder (ADHD, striato-thalamo-cortical and salience areas) and in autism spectrum disorder (ASD, salience and fronto-temporal areas) that link with core symptoms but uncorrelate with age and motion. These results replicate in an independent cohort. Downstream classification accuracy between ADHD/ASD and controls is markedly higher for CR-mCCAR compared to fusion and regression separately. CR-mCCAR can be extended to include multiple targets and multiple covariates. Overall, results demonstrate CR-mCCAR can jointly optimize for target components that correlate with the reference(s) while removing nuisance covariates. This approach can improve the meaningful detection of reliable phenotype-linked multimodal biomarkers for brain disorders.

Recently, functional magnetic resonance imaging (fMRI)-based brain networks have been shown to be an effective diagnostic tool with great potential for accurately detecting autism spectrum disorders (ASD). Meanwhile, the successful use of graph convolution networks (GCNs) methods based on fMRI information has improved the classification accuracy of ASD. However, many graph convolution-based methods do not fully utilize the topological information of the brain functional connectivity network (BFCN) or ignore the effect of non-imaging information. Therefore, we propose a hierarchical graph embedding model that leverage both the topological information of the BFCN and the non-imaging information of the subjects to improve the classification accuracy. Specifically, our model first use the Infomax Module to automatically identify embedded features in regions of interests (ROIs) in the brain. Then, these features, along with non-imaging information, is used to construct a population graph model. Finally, we design a graph convolution framework to propagate and aggregate the node features and obtain the results for ASD detection. Our model takes into account both the significance of the BFCN to individual subjects and relationships between subjects in the population graph. The model performed autism detection using the Autism Brain Imaging Data Exchange (ABIDE) dataset and obtained an average accuracy of 77.2% and an AUC of 87.2%. These results exceed those of the baseline approach. Through extensive experiments, we demonstrate the competitiveness, robustness and effectiveness of our model in aiding ASD diagnosis.

Accurate spine segmentation is crucial in clinical diagnosis and treatment of spine diseases. However, due to the complexity of spine anatomical structure, it has remained a challenging task to accurately segment spine images. Recently, the segment anything model (SAM) has achieved superior performance for image segmentation. However, generating high-quality points and boxes is still laborious for high-dimensional medical images. Meanwhile, an accurate mask is difficult to obtain. To address these issues, in this paper, we propose a multi-atlas guided SAM using multiple pseudo mask prompts for spine image segmentation, called MA-SAM. Specifically, we first design a multi-atlas prompt generation sub-network to obtain the anatomical structure prompts. More specifically, we use a network to obtain coarse mask of the input image. Then atlas label maps are registered to the coarse mask. Subsequently, a SAM-based segmentation sub-network is used to segment images. Specifically, we first utilize adapters to fine-tune the image encoder. Meanwhile, we use a prompt encoder to learn the anatomical structure prior knowledge from the multi-atlas prompts. Finally, a mask decoder is used to fuse the image and prompt features to obtain the segmentation results. Moreover, to boost the segmentation performance, different scale features from the prompt encoder are concatenated to the Upsample Block in the mask decoder. We validate our MA-SAM on the two spine segmentation tasks, including spine anatomical structure segmentation with CT images and lumbosacral plexus segmentation with MR images. Experiment results suggest that our method achieves better segmentation performance than SAM with points, boxes, and mask prompts.

The dynamic functional brain network (DFBN) inherently captures topological changes in brain connectivity pattern during activity, attracting increasing attention for detecting brain disorders. However, most current DFBN analysis methods rely on data-driven modeling and ignore crucial prior knowledge of brain structure and function, resulting in weak interpretability of models. Furthermore, effectively extracting dynamic topological features from DFBN is still a challenging issue, due to its intricate spatio-temporal features coupling. In this paper, we propose an interpretable spatio-temporal tensor graph convolutional network for DFBN analysis. Firstly, by incorporating functional and structural priors into the construction of DBFN, we develop a hierarchical DBFN representation with brain region clustering that effectively captures the spatio-temporal topology among subnetworks. Secondly, we design a tensor graph convolutional network with both intra-graph propagation and inter-graph propagation to simultaneously extract the spatio-temporal features from the hierarchical DFBN. Additionally, we derive a functional subnetwork constraint to enhance the consistency within subnetworks and the differences between subnetworks, which guides the learned features to better reflect the topology prior of the brain network. Finally, self-attention is employed to fuse the learned dynamic topological features of different subnetworks for classification. Experimental results on epilepsy, ADNI and ABIDE datasets demonstrate that our method achieves competitive diagnostic performance and offers network-level interpretability for brain disease diagnosis.

Brain imaging genetics is a widely focused topic, which has achieved the great successes in the diagnosis of complex brain disorders. In clinical practice, most existing data fusion approaches extract features from homogeneous data, neglecting the heterogeneous structural information among imaging genetic data. In addition, the number of labeled samples is limited due to the cost and time of manually labeling data. To remedy such deficiencies, in this work, we present a multimodal fusion-based hypergraph transductive learning (MFHT) for clinical diagnosis. Specifically, for each modality, we first construct a corresponding similarity graph to reflect the similarity between subjects using the label prior. Then, the multiple graph fusion approach based on theoretical convergence guarantee is designed for learning a unified graph harnessing the structure of entire data. Finally, to fully exploit the rich information of the obtained graph, a hypergraph transductive learning approach is designed to effectively capture the complex structures and high-order relationships in both labeled and unlabeled data to achieve the diagnosis results. The brain imaging genetic data of the Alzheimer's Disease Neuroimaging Initiative (ADNI) datasets are used to experimentally explore our developed method. Related results show that our method is well applied to the analysis of brain imaging genetic data, which accounts for genetics, brain imaging (region of interest (ROI) node features), and brain imaging (connectivity edge features) to boost the understanding of disease mechanism as well as improve clinical diagnosis.

Sparse views X-ray computed tomography has emerged as a contemporary technique to mitigate radiation dose. Because of the reduced number of projection views, traditional reconstruction methods can lead to severe artifacts. Recently, research studies utilizing deep learning methods has made promising progress in removing artifacts for Sparse-View Computed Tomography (SVCT). However, given the limitations on the generalization capability of deep learning models, current methods usually train models on fixed sampling rates, affecting the usability and flexibility of model deployment in real clinical settings. To address this issue, our study proposes a adaptive reconstruction method to achieve high-performance SVCT reconstruction at various sampling rate. Specifically, we design a novel imaging degradation operator in the proposed sampling diffusion model for SVCT (CT-SDM) to simulate the projection process in the sinogram domain. Thus, the CT-SDM can gradually add projection views to highly undersampled measurements to generalize the full-view sinograms. By choosing an appropriate starting point in diffusion inference, the proposed model can recover the full-view sinograms from various sampling rate with only one trained model. Experiments on several datasets have verified the effectiveness and robustness of our approach, demonstrating its superiority in reconstructing high-quality images from sparse-view CT scans across various sampling rates.

Segment Anything Model (SAM) is a foundational image segmentation model, which shows superior performance for natural image segmentation tasks. Several SAM-based medical image segmentations have been proposed. However, these SAM-based medical image segmentation methods heavily depend on prior manual guidance involving points, boxes, and coarse-grained masks, which lack adaptability and flexibility. Moreover, the inherent challenge of edge blurring in medical images is critical, as it directly affects the quality of segmentation. To address these challenges, we propose an uncertainty-driven edge prompt generation network for medical image segmentation, called UDEG-Net. Specifically, to better adapt to medical image segmentation, we fine-tune the encoder by using Low-Rank Adaptation (LoRA) technology to enhance the encoder's learning capability and capture enriched medical image features. Furthermore, to overcome the limitations of interactive prompts, we develop an auto edge prompt generator to generate edge prompt information and further enhance the structural representation. Finally, to focus on the high-uncertainty edge areas, we introduce an evidence-based uncertainty estimation and a progressive uncertainty-driven loss to drive the auto edge prompt generator to yield robust edge prompt information and reliable segmentation results. Experimental results on three public datasets and one private dataset show that our UDEG-Net outperforms the state-of-the-art medical image segmentation methods.

Multi-task learning (MTL) has become a research hotspot for the analysis of whole-slide histopathological images (WSIs) since it can capture the shared representations of different tasks for the improvement of individual tasks. However, the shared representations learned by MTL are always dominated by the tasks appearing in the training set that is difficult to directly apply it on the unseen (new) tasks, especially when the unseen tasks are significantly different from the known tasks. To address the above issues, we develop a Task-Agnostic Feature-Learner (TAFL) for efficient adaptation to unseen clinical tasks, which can leverage useful image information from the existing tasks for new clinical trials with minimal task-specific modifications. Specifically, we firstly develop a neural architecture search (NAS) module that can design the network architectures of TAFL automatically. Then, a novel task-level meta-learning algorithm is developed to extract efficient and universal information from the known tasks for improving the prediction performance on the unseen tasks. We evaluate our method on three publicly available datasets derived from The Cancer Genome Atlas (TCGA) for various clinical prediction tasks (i.e., staging, cancer subtyping and survival prediction), and the experimental results indicate that our TAFL can effectively adapt to unseen tasks with better prediction performance.

Functional magnetic resonance imaging (fMRI), as a non-invasive method to reveal brain function alterations, frequently yields time series with unequal lengths in real-world scenarios, which may arise from factors such as motion artifacts, participant state, and differing scan protocols. This variability conflicts with the traditional methods relying on isometric inputs, which poses a significant challenge for the downstream applications such as brain age prediction. To address this challenge, we introduced Gaussian Process Regression (GPR) to normalize the length of time series and proposed split-channel residual convolution (SC) and self-attention mechanisms (SA) to perform brain age estimation, called GPR-SCSANet. Results showed that the proposed framework, GPR-SCSANet, is able to fully utilize the inherent information and learn richer feature representations from unequal-length fMRI time courses, which significantly improved the prediction accuracy across 3 brain atlases and 5 prediction models. The results demonstrated the effectiveness and robustness of the proposed GPR-SCSANet, showcasing the potential for broader applications in brain age prediction task.

The ingestible capsule robot has shown significant promise for non-invasive diagnosis and drug delivery within the gastrointestinal tract. Locomotion control of the capsule robot requires real-time, accurate localization and angle estimation from ultrasound images. This paper introduces a novel CNN model with spatially adaptive Gaussian distribution, which can simultaneously finish the localization and angle estimation of capsule robots in US images. Specifically, we propose a robot-adaptive label assignment strategy using an elliptical Gaussian to better suit the shape and orientation of the capsule robot. Additionally, we employ a novel bounding box representation for capsule robot localization to encode predictions from the CNN model. Experimental results demonstrate that our method finishes real-time localization and angle estimation of capsule robots at approximately 48 frames per second. Moreover, our approach achieves a small localization error of 0.16 mm and a high angle estimation accuracy of 99.85%. The method developed, utilizing spatially adaptive Gaussian distribution, holds practical significance in achieving accurate and real-time position and angle feedback for the locomotion control of capsule robots.

Understanding neurological manifestations and their genetic architectures are important for exploring the etiology and pathology of brain disorders. Multimodal neuroimaging data carry complementary information and are known to exhibit shared and specific characteristics from different perspectives. Hence, exploring modality-shared and modality-specific imaging features as well as their genetic underpinnings is a challenging but beneficial task. Unfortunately, this issue has been largely unexploited. In this paper, to fill this gap, we propose a fresh and straightforward insight, referred as Multimodality-Disentangled Phenotype-Genotype Correlation approach (MDPGC). Specifically, we design a unified framework for exploring the multimodality-disentangled characteristics of image-based phenotypes, and further detect genetic variants associated with the disorder using modality-shared and modality-specific biomarkers as intermediate phenotypes. Extensive experimental results on Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset reveal that our method attains superior correlation coefficients compared to state-of-the-art methods, and at the same time provided excellent interpretability. In addition, the subsequent analysis demonstrates that MDPGC successfully identifies different types of characteristics of imaging phenotypes and reveals relevant genetic variations. These findings not only contribute to AD diagnosis but also help better understand the pathological and pathogenic mechanisms of brain disorders.

The recent advance of deep learning technology brings the possibility of assisting the pathologist to predict the patients' survival from whole-slide pathological images (WSIs). However, most of the prevalent methods only worked on the sampled patches in specifically or randomly selected tumor areas of WSIs, which has very limited capability to capture the complex interactions between tumor and its surrounding micro-environment components. As a matter of fact, tumor is supported and nurtured in the heterogeneous tumor micro-environment(TME), and the detailed analysis of TME and their correlation with tumors are important to in-depth analyze the mechanism of cancer development. In this paper, we considered the spatial interactions among tumor and its two major TME components (i.e., lymphocytes and stromal fibrosis) and presented a Tumor Micro-environment Interactions Guided Graph Learning (TMEGL) algorithm for the prognosis prediction of human cancers. Specifically, we firstly selected different types of patches as nodes to build graph for each WSI. Then, a novel TME neighborhood organization guided graph embedding algorithm was proposed to learn node representations that can preserve their topological structure information. Finally, a Gated Graph Attention Network is applied to capture the survival-associated intersections among tumor and different TME components for clinical outcome prediction. We tested TMEGL on three cancer cohorts derived from The Cancer Genome Atlas (TCGA), and the experimental results indicated that TMEGL not only outperforms the existing WSI-based survival analysis models, but also has good explainable ability for survival prediction.

MOTIVATION: Nanopore sequencing is a new macromolecular recognition and perception technology that enables high-throughput sequencing of DNA, RNA, even protein molecules. The sequences generated by nanopore sequencing span a large time frame, and the labor and time costs incurred by traditional analysis methods are substantial. Recently, research on nanopore data analysis using machine learning algorithms has gained unceasing momentum, but there is often a significant gap between traditional and deep learning methods in terms of classification results. To analyze nanopore data using deep learning technologies, measures such as sequence completion and sequence transformation can be employed. However, these technologies do not preserve the local features of the sequences. To address this issue, we propose a sequence-to-image (S2I) module that transforms sequences of unequal length into images. Additionally, we propose the Transformer-based T-S2Inet model to capture the important information and improve the classification accuracy. RESULTS: Quantitative and qualitative analysis shows that the experimental results have an improvement of around 2% in accuracy compared to previous methods. The proposed method is adaptable to other nanopore platforms, such as the Oxford nanopore. It is worth noting that the proposed method not only aims to achieve the most advanced performance, but also provides a general idea for the analysis of nanopore sequences of unequal length. AVAILABILITY AND IMPLEMENTATION: The main program is available at https://github.com/guanxiaoyu11/S2Inet.

Model intellectual property (IP) protection has gained attention due to the significance of safeguarding intellectual labor and computational resources. Ensuring IP safety for trainers and owners is critical, especially when ownership verification and applicability authorization are required. A notable approach involves preventing the transfer of well-trained models from authorized to unauthorized domains. We introduce a novel Compact Un-transferable Pyramid Isolation Domain (CUPI-Domain) which serves as a barrier against illegal transfers from authorized to unauthorized domains. Inspired by human transitive inference, the CUPI-Domain emphasizes distinctive style features of the authorized domain, leading to failure in recognizing irrelevant private style features on unauthorized domains. To this end, we propose CUPI-Domain generators, which select features from both authorized and CUPI-Domain as anchors. These generators fuse the style features and semantic features to create labeled, style-rich CUPI-Domain. Additionally, we design external Domain-Information Memory Banks (DIMB) for storing and updating labeled pyramid features to obtain stable domain class features and domain class-wise style features. Based on the proposed whole method, the novel style and discriminative loss functions are designed to effectively enhance the distinction in style and discriminative features between authorized and unauthorized domains. We offer two solutions for utilizing CUPI-Domain based on whether the unauthorized domain is known: target-specified CUPI-Domain and target-free CUPI-Domain. Comprehensive experiments on various public datasets demonstrate the effectiveness of our CUPI-Domain approach with different backbone models, providing an efficient solution for model intellectual property protection.

Multi-modal emotion recognition has attracted increasing attention in human-computer interaction, as it extracts complementary information from physiological and behavioral features. Compared to single modal approaches, multi-modal fusion methods are more susceptible to uncertainty in emotion recognition, such as heterogeneity and inconsistent predictions across different modalities. Previous multi-modal approaches ignore systematic modeling of uncertainty in fusion and revelation of dynamic variations in emotion process. In this paper, we propose a dynamic confidence-aware fusion network for robust recognition of heterogeneous emotion features, including electroencephalogram (EEG) and facial expression. First, we develop a self-attention based multi-channel LSTM network to preliminarily align the heterogeneous emotion features. Second, we propose a confidence regression network to estimate true class probability (TCP) on each modality, which helps explore the uncertainty at modality level. Then, different modalities are weighted fused according to above two types of uncertainty. Finally, we adopt self-paced learning (SPL) mechanism to further improve the model robustness by alleviating negative effect from the hard learning samples. The experimental results on several multi-modal emotion datasets demonstrate the proposed method outperforms the state-of-the-art methods in emotion recognition performance and explicitly reveals the dynamic variation of emotion with uncertainty estimation. Our code is available at:

Machine learning techniques have become increasingly important in analyzing single-cell RNA and identifying cell types, providing valuable insights into cellular development and disease mechanisms. However, the presence of batch effects poses major challenges in scRNA-seq analysis due to data distribution variation across batches. Although several batch effect mitigation algorithms have been proposed, most of them focus only on the correlation of local structure embeddings, ignoring global distribution matching and discriminative feature representation in batch correction. In this paper, we proposed the discriminative domain adaption network (D2AN) for joint batch effects correction and type annotation with single-cell RNA-seq. Specifically, we first captured the global low-dimensional embeddings of samples from the source and target domains by adversarial domain adaption strategy. Second, a contrastive loss is developed to preliminarily align the source domain samples. Moreover, the semantic alignment of class centroids in the source and target domains is achieved for further local alignment. Finally, a self-paced learning mechanism based on inter-domain loss is adopted to gradually select samples with high similarity to the target domain for training, which is used to improve the robustness of the model. Experimental results demonstrated that the proposed method on multiple real datasets outperforms several state-of-the-art methods.

Brain region-of-interest (ROI) segmentation with magnetic resonance (MR) images is a basic prerequisite step for brain analysis. The main problem with using deep learning for brain ROI segmentation is the lack of sufficient annotated data. To address this issue, in this paper, we propose a simple multi-atlas supervised contrastive learning framework (MAS-CL) for brain ROI segmentation with MR images in an end-to-end manner. Specifically, our MAS-CL framework mainly consists of two steps, including 1) a multi-atlas supervised contrastive learning method to learn the latent representation using a limited amount of voxel-level labeling brain MR images, and 2) brain ROI segmentation based on the pre-trained backbone using our MSA-CL method. Specifically, different from traditional contrastive learning, in our proposed method, we use multi-atlas supervised information to pre-train the backbone for learning the latent representation of input MR image, i.e., the correlation of each sample pair is defined by using the label maps of input MR image and atlas images. Then, we extend the pre-trained backbone to segment brain ROI with MR images. We perform our proposed MAS-CL framework with five segmentation methods on LONI-LPBA40, IXI, OASIS, ADNI, and CC359 datasets for brain ROI segmentation with MR images. Various experimental results suggested that our proposed MAS-CL framework can significantly improve the segmentation performance on these five datasets.

Predicting individual behavior is a crucial area of research in neuroscience. Graph Neural Networks (GNNs), as powerful tools for extracting graph-structured features, are increasingly being utilized in various functional connectivity (FC) based behavioral prediction tasks. However, current predictive models primarily focus on enhancing GNNs' ability to extract features from FC networks while neglecting the importance of upstream individual network construction quality. This oversight results in constructed functional networks that fail to adequately represent individual behavioral capacity, thereby affecting the subsequent prediction accuracy. To address this issue, we proposed a new GNN-based behavioral prediction framework, named Dual Multi-Hop Graph Convolutional Network (D-MHGCN). Through the joint training of two GCNs, this framework integrates individual functional network construction and behavioral prediction into a unified optimization model. It allows the model to dynamically adjust the individual functional cortical parcellation according to the downstream tasks, thus creating task-aware, individual-specific FCNs that largely enhance its ability to predict behavior scores. Additionally, we employed multi-hop graph convolution layers instead of traditional single-hop methods in GCN to capture complex hierarchical connectivity patterns in brain networks. Our experimental evaluations, conducted on the large, public Human Connectome Project dataset, demonstrate that our proposed method outperforms existing methods in various behavioral prediction tasks. Moreover, it produces more functionally homogeneous cortical parcellation, showcasing its practical utility and effectiveness. Our work not only enhances the accuracy of individual behavioral prediction but also provides deeper insights into the neural mechanisms underlying individual differences in behavior.

Fatigue driving is a significant cause of road traffic accidents and associated casualties. Automatic assessment of driver drowsiness by monitoring electroencephalography (EEG) signals offer a more objective way to improve driving safety. However, most existing measures are based on multi-channel EEG signals, which are more difficult to apply in practical scenarios as it usually lacks better portability and comfort. In addition, due to the relatively parsimonious and non-stationary characteristics, it is still challenging to effectively accomplish drowsiness recognition by exploiting single-channel EEG signals alone. To this end, we propose a novel temporal-frequential attentional convolutional neural network (TFAC-Net) to take full advantage of spectral-temporal features for single-channel EEG driver drowsiness recognition. Specifically, to capture the potentially valuable information contained in single-channel EEG, the continuous wavelet transform is first employed to generate a corresponding spectral-temporal representation. Then, the temporal-frequential attention mechanism is adopted to reveal critical time-frequency regions in terms of the driver’s mental state. Finally, an adaptive feature fusion module is considered to recalibrate and integrate the most relevant feature channels for final prediction. Extensive experimental results on a widely used public EEG driving dataset demonstrate that the TFAC-Net approach is superior to the state-of-the-art methods, and could discover some discriminative temporal-frequential regions. Moreover, this study also sheds light on the development of portable EEG devices and practical driver drowsiness recognition.

Dynamic contrast-enhanced ultrasound (CEUS) imaging can reflect the microvascular distribution and blood flow perfusion, thereby holding clinical significance in distinguishing between malignant and benign thyroid nodules. Notably, CEUS offers a meticulous visualization of the microvascular distribution surrounding the nodule, leading to an apparent increase in tumor size compared to gray-scale ultrasound (US). In the dual-image obtained, the lesion size enlarged from gray-scale US to CEUS, as the microvascular appeared to be continuously infiltrating the surrounding tissue. Although the infiltrative dilatation of microvasculature remains ambiguous, sonographers believe it may promote the diagnosis of thyroid nodules. We propose a deep learning model designed to emulate the diagnostic reasoning process employed by sonographers. This model integrates the observation of microvascular infiltration on dynamic CEUS, leveraging the additional insights provided by gray-scale US for enhanced diagnostic support. Specifically, temporal projection attention is implemented on time dimension of dynamic CEUS to represent the microvascular perfusion. Additionally, we employ a group of confidence maps with flexible Sigmoid Alpha Functions to aware and describe the infiltrative dilatation process. Moreover, a self-adaptive integration mechanism is introduced to dynamically integrate the assisted gray-scale US and the confidence maps of CEUS for individual patients, ensuring a trustworthy diagnosis of thyroid nodules. In this retrospective study, we collected a thyroid nodule dataset of 282 CEUS videos. The method achieves a superior diagnostic accuracy and sensitivity of 89.52% and 94.75%, respectively. These results suggest that imitating the diagnostic thinking of sonographers, encompassing dynamic microvascular perfusion and infiltrative expansion, proves beneficial for CEUS-based thyroid nodule diagnosis.

Brain networks, describing the functional or structural interactions of brain with graph theory, have been widely used for brain imaging analysis. Currently, several network representation methods have been developed for describing and analyzing brain networks. However, most of these methods ignored the valuable weighted information of the edges in brain networks. In this paper, we propose a new representation method (i.e., ordinal pattern tree) for brain network analysis. Compared with the existing network representation methods, the proposed ordinal pattern tree (OPT) can not only leverage the weighted information of the edges but also express the hierarchical relationships of nodes in brain networks. On OPT, nodes are connected by ordinal edges which are constructed by using the ordinal pattern relationships of weighted edges. We represent brain networks as OPTs and further develop a new graph kernel called optimal transport (OT) based ordinal pattern tree (OT-OPT) kernel to measure the similarity between paired brain networks. In OT-OPT kernel, the OT distances are used to calculate the transport costs between the nodes on the OPTs. Based on these OT distances, we use exponential function to calculate OT-OPT kernel which is proved to be positive definite. To evaluate the effectiveness of the proposed method, we perform classification and regression experiments on ADHD-200, ABIDE and ADNI datasets. The experimental results demonstrate that our proposed method outperforms the state-of-the-art graph methods in the classification and regression tasks.

With the remarkable success of digital histopathology and the deep learning technology, many whole-slide pathological images (WSIs) based deep learning models are designed to help pathologists diagnose human cancers. Recently, rather than predicting categorical variables as in cancer diagnosis, several deep learning studies are also proposed to estimate the continuous variables such as the patients' survival or their transcriptional profile. However, most of the existing studies focus on conducting these predicting tasks separately, which overlooks the useful intrinsic correlation among them that can boost the prediction performance of each individual task. In addition, it is sill challenge to design the WSI-based deep learning models, since a WSI is with huge size but annotated with coarse label. In this study, we propose a general multi-instance multi-task learning framework (HistMIMT) for multi-purpose prediction from WSIs. Specifically, we firstly propose a novel multi-instance learning module (TMICS) considering both common and specific task information across different tasks to generate bag representation for each individual task. Then, a soft-mask based fusion module with channel attention (SFCA) is developed to leverage useful information from the related tasks to help improve the prediction performance on target task. We evaluate our method on three cancer cohorts derived from the Cancer Genome Atlas (TCGA). For each cohort, our multi-purpose prediction tasks range from cancer diagnosis, survival prediction and estimating the transcriptional profile of gene TP53. The experimental results demonstrated that HistMIMT can yield better outcome on all clinical prediction tasks than its competitors.

Dynamic brain network has the advantage over static brain network in characterizing the variation pattern of functional brain connectivity, and it has attracted increasing attention in brain disease diagnosis. However, most of the existing dynamic brain networks analysis methods rely on extracting features from independent brain networks divided by sliding windows, making them hard to reveal the high-order dynamic evolution laws of functional brain networks. Additionally, they cannot effectively extract the spatio-temporal topology features in dynamic brain networks. In this paper, we propose to use optimal transport (OT) theory to capture the topology evolution of the dynamic brain networks, and develop a multi-channel spatio-temporal graph convolutional network that collaboratively extracts the temporal and spatial features from the evolution networks. Specifically, we first adaptively evaluate the graph hubness of brain regions in the brain network of each time window, which comprehensively models information transmission among multiple brain regions. Second, the hubness propagation information across adjacent time windows is captured by optimal transport, describing high-order topology evolution of dynamic brain networks. Moreover, we develop a spatio-temporal graph convolutional network with attention mechanism to collaboratively extract the intrinsic temporal and spatial topology information from the above networks. Finally, the multi-layer perceptron is adopted for classifying the dynamic brain network. The extensive experiment on the collected epilepsy dataset and the public ADNI dataset show that our proposed method not only outperforms several state-of-the-art methods in brain disease diagnosis, but also reveals the key dynamic alterations of brain connectivities between patients and healthy controls.

Adversarial training (AT) has shown excellent high performance in defending against adversarial examples. Recent studies demonstrate that examples are not equally important to the final robustness of models during AT, that is, the so-called hard examples that can be attacked easily exhibit more influence than robust examples on the final robustness. Therefore, guaranteeing the robustness of hard examples is crucial for improving the final robustness of the model. However, defining effective heuristics to search for hard examples is still difficult. In this article, inspired by the information bottleneck (IB) principle, we uncover that an example with high mutual information of the input and its associated latent representation is more likely to be attacked. Based on this observation, we propose a novel and effective adversarial training method (InfoAT). InfoAT is encouraged to find examples with high mutual information and exploit them efficiently to improve the final robustness of models. Experimental results show that InfoAT achieves the best robustness among different datasets and models in comparison with several state-of-the-art methods.

Medical image registration is a crucial preprocessing step in medical image processing. Due to the potential impact of treatments and disease progression on patients' organ geometry, such as in magnetic resonance imaging (MRI) and computed tomography (CT) images, medical image registration is of significant importance for cancer diagnosis, treatment planning, and therapy. Existing methods usually adopt convolutional neural networks and Transformer frameworks, but still face challenges to effectively solve both tissue robustness and registration accuracy. Therefore, we propose TSM, a hybrid Transformer-Convolution model, for non-rigid registration of volumetric medical multi-tissue images. We perform dynamic global filtering convolution in the frequency domain and multi-scale parallel convolution, capturing local tissue structure information; meanwhile, we use a special attention combination mechanism to obtain semantic associations among tissue structures. We evaluate our method on the publicly available LPBA40 and EMPIRE10 challenge datasets. With comparison of the state-of-the-arts, we improve the dice score by 1.2% on the LPBA40 dataset and by 3% on the EMPIRE10 dataset, achieving the best registration results.

As scientific and technological advancements result from human intellectual labor and computational costs, protecting model intellectual property (IP) has become increasingly important to encourage model creators and owners. Model IP protection involves preventing the use of well-trained models on unauthorized domains. To address this issue, we propose a novel approach called Compact Un-Transferable Isolation Domain (CUTI-domain), which acts as a barrier to block illegal transfers from authorized to unauthorized domains. Specifically, CUTI-domain blocks crossdomain transfers by highlighting the private style features of the authorized domain, leading to recognition failure on unauthorized domains with irrelevant private style features. Moreover, we provide two solutions for using CUTI-domain depending on whether the unauthorized domain is known or not: target-specified CUTI-domain and target-free CUTI-domain. Our comprehensive experimental results on four digit datasets, CIFAR10 & STL10, and VisDA-2017 dataset demonstrate that CUTI-domain can be easily implemented as a plug-and-play module with different backbones, providing an efficient solution for model IP protection.

Emotion recognition based on electroencephalography (EEG) has attracted significant attention and achieved considerable advances in the fields of affective computing and human-computer interaction. However, most existing studies ignore the coupling and complementarity of complex spatiotemporal patterns in EEG signals. Moreover, how to exploit and fuse crucial discriminative aspects in high redundancy and low signal-to-noise ratio EEG signals remains a great challenge for emotion recognition. In this paper, we propose a novel attention-based spatial-temporal dual-stream fusion network, named ASTDF-Net, for EEG-based emotion recognition. Specifically, ASTDF-Net comprises three main stages: first, the collaborative embedding module is designed to learn a joint latent subspace to capture the coupling of complicated spatiotemporal information in EEG signals. Second, stacked parallel spatial and temporal attention streams are employed to extract the most essential discriminative features and filter out redundant task-irrelevant factors. Finally, the hybrid attention-based feature fusion module is proposed to integrate significant features discovered from the dual-stream structure to take full advantage of the complementarity of the diverse characteristics. Extensive experiments on two publicly available emotion recognition datasets indicate that our proposed approach consistently outperforms state-of-the-art methods.

MOTIVATION: As the third-generation sequencing technology, nanopore sequencing has been used for high-throughput sequencing of DNA, RNA, and even proteins. Recently, many studies have begun to use machine learning technology to analyze the enormous data generated by nanopores. Unfortunately, the success of this technology is due to the extensive labeled data, which often suffer from enormous labor costs. Therefore, there is an urgent need for a novel technology that can not only rapidly analyze nanopore data with high-throughput, but also significantly reduce the cost of labeling. To achieve the above goals, we introduce active learning to alleviate the enormous labor costs by selecting the samples that need to be labeled. This work applies several advanced active learning technologies to the nanopore data, including the RNA classification dataset (RNA-CD) and the Oxford Nanopore Technologies barcode dataset (ONT-BD). Due to the complexity of the nanopore data (with noise sequence), the bias constraint is introduced to improve the sample selection strategy in active learning. Results: The experimental results show that for the same performance metric, 50% labeling amount can achieve the best baseline performance for ONT-BD, while only 15% labeling amount can achieve the best baseline performance for RNA-CD. Crucially, the experiments show that active learning technology can assist experts in labeling samples, and significantly reduce the labeling cost. Active learning can greatly reduce the dilemma of difficult labeling of high-capacity nanopore data. We hope active learning can be applied to other problems in nanopore sequence analysis. AVAILABILITY AND IMPLEMENTATION: The main program is available at https://github.com/guanxiaoyu11/AL-for-nanopore. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Deep Neural Network (DNN) models are usually trained sequentially from one layer to another, which causes forward, backward and update locking problems, leading to poor performance in terms of training time. The existing parallel strategies to mitigate these problems provide suboptimal runtime performance. In this work, we have proposed a novel layer-wise partitioning and merging, forward and backward pass parallel framework to provide better training performance. The novelty of the proposed work consists of 1) a layer-wise partition and merging model which can minimise communication overhead between devices without the memory cost of existing strategies during the training process; 2) a forward pass and backward pass parallelisation to address the update locking problem and minimise the total training cost. The experimental evaluation on real use cases shows that the proposed method outperforms the state-of-the-art approaches in terms of training speed; and achieves almost linear speedup without compromising the accuracy performance of the non-parallel approach.

Brain network , which characterizes the functional and structural interactions of brain regions with graph theory, has been widely utilized to diagnose brain diseases, such as autism spectrum disorder (ASD). It is a challenge to measure the network (or graph) similarity in brain network analysis . Graph kernel (i.e., kernel defined on graphs) offers an efficient tool for measuring the similarity of paired brain networks and yields the excellent classification performance in brain disease diagnosis. However, most of the existing graph kernels neglected the hierarchical architecture information of brain networks. To address this problem, in this paper, we propose an optimal transport based pyramid graph kernel for measuring brain network similarity and then apply it to brain disease classification. The main idea is to transform brain networks into pyramid structures, which reflect the hierarchical architecture information of the brain network with multi-resolution histograms. The optimal transport distance in pyramid structures is calculated for measuring transport costs between paired brain networks. Finally, the optimal transport based pyramid graph kernel is computed based on this optimal transport distance. To evaluate the effectiveness of the proposed optimal transport based pyramid graph kernel, the extensive experiments are performed in functional magnetic resonance imaging data of brain disease from the Autism Brain Imaging Data Exchange database. The experimental results show that our proposed optimal transport based pyramid graph kernel outperforms the state-of-the-art methods in ASD classification tasks .

The multi-site approach has attracted increasing attention in brain disease diagnosis, because it can improve the prediction performance by integrating sample information from different medical institutions. However, its training procedure requires the transmission of subject's original images or features among sites, which may cause privacy disclosure. In this paper, we propose a self-supervised federated adaptation (S2FA) framework for robust multi-site prediction, which can reduce the risk of privacy disclosure. As far as we know, it is the first work to investigate the cross-site brain disease diagnosis, which trains model on source sites and tests on target site, often occurring in clinical practice. Firstly, we implement a decentralized federated optimization strategy, by which each site communicates model parameters periodically. Secondly, we construct an auxiliary self-supervised model for target site through transferring knowledge from source sites with self-paced learning. Then, a hash mapping is proposed to encode the target feature, simultaneously reducing the risk of privacy information disclosure and alleviating data heterogeneity among sites. Finally, we achieve the cross-site prediction by weighted federated source model and auxiliary target model. Experimental results on multi-site datasets show that the proposed S2FA can accurately identify brain disease. Our codes are available athttps://github.com/nuaayqm/S2FA.

Multi-center analysis has attracted increasing attention in brain disease diagnosis, because it provides effective approaches to improve disease diagnostic performance by making use of the information from different centers. However, in practical multi-center applications, data uncertainty is more common than that in single center, which brings challenge to robust modeling of diagnosis. In this article, we proposed a multi-discriminator active adversarial network (MDAAN) to alleviate the uncertainties at the center, feature, and label levels for multi-center brain disease diagnosis. First, we extract the latent invariant representation of the source center and target center to reduce domain shift by adversarial learning strategy. Second, the proposed method adaptively evaluates the contribution of different source centers in fusion by measuring data distribution difference between source and target center. Moreover, only the hard learning samples in target center are identified to label with low sample annotation cost. Finally, we treat the selected samples as the auxiliary domain to alleviate the negative transfer and improve the robustness of the multi-center model. We extensively compare the proposed approach with several state-of-the-art multi-center methods on the five-center schizophrenia dataset, and the results demonstrate that our method is superior to the previous methods in identifying brain disease.

In recent years, machine learning has gained increasing traction in the study of molecules, enabling researchers to tackle challenging tasks including molecular property prediction and drug design.Consequently, there remains an open challenge to develop a neural network architecture that can make use of extensive amounts of unlabeled data for training while still providing competitive results in various molecular property prediction tasks. To address this challenge, we propose a Molecule Graph Contrastive Learning approach based on the Transformer framework (T-MGCL). Our approach involves expanding numerous unsupervised molecular graphs and using a contrast estimator to ensure consistency among various graph expansions of the same molecule. Transformer framework is employed to consider the distance between atoms and molecular graph attributes, thereby accounting for structural information that may be overlooked by traditional graph neural networks. Our experimental results demonstrate that the T-MGCL model outperforms other models in several molecular property prediction tasks. Additionally, we observe that the attention weight learned by T-MGCL can be easily explained from a chemical perspective.

Contactless palmprint recognition, an emerging biometric technology, has attracted increasing attention due to its noninvasive and high practicability characteristics. Although it is naturally suitable for mobile application scenarios, the following two challenges severely limit its recognition performance: 1) the inconsistency in acquisition devices used in training and testing, and 2) many subjects are unable to be imaged on each device, resulting in incomplete data problems. To address these issues, we propose a self-paced CycleGAN with self-attention modules, which simultaneously synthesizes missing data and alleviates the influence of different imaging devices. Specifically, we develop CycleGAN with self-attention modules to generate missing training data by effectively mining the structural correlation among samples while capturing the cross-domain features. Furthermore, a self-paced learning strategy, which is a human cognitive-driven learning mechanism, is used to guide learning the robust cross-domain feature representation and recognition model, by which the relatively easy learning samples are gradually involved in the training process. To verify the effectiveness of the proposed method, we conduct experiments on contactless palmprint datasets collected using different smartphones. The results show that our approach outperforms state-of-the-art methods in classifying contactless palmprint images.

As an emerging biometric technology, multi-spectral palmprint recognition has attracted increasing attention in security due to its high accuracy and ease of use. Compared to single spectral case, multi-spectral palmprint model is more susceptible to the attack of adversarial examples. However, the previous adversarial example attack approaches cannot generate the most aggressive adversarial examples for multi-spectral palmprint recognition. In addition, most of them are dependent on the explicit architecture or need time-consuming queries about the network to be attacked, which significantly limits their application in the field of security. To solve the above problems, in this paper, we proposed the multi-spectral palmprints joint attack and defense framework based on multi-view adversarial examples learning. First, we respectively capture the multi-view deep common feature space for the different spectra and the discriminative feature space across the different subjects. Second, we introduce perturbation in the deep common space to achieve adversarial multi-spectral palmprints with gradient propagation. In addition, we pursue the manifold of the difference space and use it to suppress the discriminability of the recognition model with adversarial region theory. Finally, the generated adversarial examples are fed into the training model to enhance the robustness of the recognition algorithm. The experimental results on multi-spectral palmprint dataset demonstrate that the proposed multi-view joint attack approach is superior to the state-of-the-art adversarial example attack methods in attack accuracy and transferability. Moreover, the defense strategy with the adversarial examples by our method can significantly promote the robustness of multi-spectral palmprint recognition methods.

Dynamic contrast-enhanced ultrasound (CEUS) imaging has been widely applied in lesion detection and characterization, due to its offered real-time observation of microvascular perfusion. Accurate lesion segmentation is of great importance to the quantitative and qualitative perfusion analysis. In this paper, we propose a novel dynamic perfusion representation and aggregation network (DpRAN) for the automatic segmentation of lesions using dynamic CEUS imaging. The core challenge of this work lies in enhancement dynamics modeling of various perfusion areas. Specifically, we divide enhancement features into the two scales: short-range enhancement patterns and long-range evolution tendency. To effectively represent real-time enhancement characteristics and aggregate them in a global view, we introduce the perfusion excitation (PE) gate and cross-attention temporal aggregation (CTA) module, respectively. Different from the common temporal fusion methods, we also introduce an uncertainty estimation strategy to assist the model to locate the critical enhancement point first, in which a relatively distinguished enhancement pattern is displayed. The segmentation performance of our DpRAN method is validated on our collected CEUS datasets of thyroid nodules. We obtain the mean dice coefficient (DSC) and intersection of union (IoU) of 0.794 and 0.676, respectively. Superior performance demonstrates its efficacy to capture distinguished enhancement characteristics for lesion recognition.

Accurate and rapid detection of COVID-19 pneumonia is crucial for optimal patient treatment. Chest X-Ray (CXR) is the first-line imaging technique for COVID-19 pneumonia diagnosis as it is fast, cheap and easily accessible. Currently, many deep learning (DL) models have been proposed to detect COVID-19 pneumonia from CXR images. Unfortunately, these deep classifiers lack the transparency in interpreting findings, which may limit their applications in clinical practice. The existing explanation methods produce either too noisy or imprecise results, and hence are unsuitable for diagnostic purposes. In this work, we propose a novel explainable CXR deep neural Network (CXR-Net) for accurate COVID-19 pneumonia detection with an enhanced pixel-level visual explanation using CXR images. An Encoder-Decoder-Encoder architecture is proposed, in which an extra encoder is added after the encoder-decoder structure to ensure the model can be trained on category samples. The method has been evaluated on real world CXR datasets from both public and private sources, including healthy, bacterial pneumonia, viral pneumonia and COVID-19 pneumonia cases. The results demonstrate that the proposed method can achieve a satisfactory accuracy and provide fine-resolution activation maps for visual explanation in the lung disease detection. Compared to current state-of-the-art visual explanation methods, the proposed method can provide more detailed, high-resolution, visual explanation for the classification results. It can be deployed in various computing environments, including cloud, CPU and GPU environments. It has a great potential to be used in clinical practice for COVID-19 pneumonia diagnosis.

Accurate ultrasound (US) image segmentation is crucial for the screening and diagnosis of diseases. However, it faces two significant challenges: 1) pixel-level annotation is a time-consuming and laborious process; 2) the presence of shadow artifacts leads to missing anatomy and ambiguous boundaries, which negatively impact reliable segmentation results. To address these challenges, we propose a novel semi-supervised shadow aware network with boundary refinement (SABR-Net). Specifically, we add shadow imitation regions to the original US, and design shadow-masked transformer blocks to perceive missing anatomy of shadow regions. Shadow-masked transformer block contains an adaptive shadow attention mechanism that introduces an adaptive mask, which is updated automatically to promote the network training. Additionally, we utilize unlabeled US images to train a missing structure inpainting path with shadow-masked transformer, which further facilitates semi-supervised segmentation. Experiments on two public US datasets demonstrate the superior performance of the SABR-Net over other state-of-the-art semi-supervised segmentation methods. In addition, experiments on a private breast US dataset prove that our method has a good generalization to clinical small-scale US datasets.

Survival analysis is to estimate the survival time for an individual or a group of patients, which is a valid solution for cancer treatments. Recent studies suggested that the integrative analysis of histopathological images and genomic data can better predict the survival of cancer patients than simply using single bio-marker, for different bio-markers may provide complementary information. However, for the given multi-modal data that may contain irrelevant or redundant features, it is still challenge to design a distance metric that can simultaneously discover significant features and measure the difference of survival time among different patients. To solve this issue, we propose a Feature-Aware Multi-modal Metric Learning method (FAM3L), which not only learns the metric for distance constraints on patients' survival time, but also identifies important images and genomic features for survival analysis. Specifically, for each modality of data, we firstly design one feature-aware metric that can be decoupled into a traditional distance metric and a diagonal weight for important feature identification. Then, in order to explore the complex correlation across multiple modality data, we apply Hilbert-Schmidt Independence Criterion (HSIC) to jointly learn multiple metrics. Finally, based on the learned distance metrics, we apply the Cox proportional hazards model for prognosis prediction. We evaluate the performance of our proposed FAM3L method on three cancer cohorts derived from The Cancer Genome Atlas (TCGA), the experimental results demonstrate that our method can not only achieve superior performance for cancer prognosis, but also identify meaningful image and genomic features correlating strongly with cancer survival.

The tumor-infiltrating lymphocytes (TILs) and its correlation with tumors have shown significant values in the development of cancers. Many observations indicated that the combination of the whole-slide pathological images (WSIs) and genomic data can better characterize the immunological mechanisms of TILs. However, the existing image-genomic studies evaluated the TILs by the combination of pathological image and single-type of omics data (e.g., mRNA), which is difficulty in assessing the underlying molecular processes of TILs holistically. Additionally, it is still very challenging to characterize the intersections between TILs and tumor regions in WSIs and the high dimensional genomic data also brings difficulty for the integrative analysis with WSIs. Based on the above considerations, we proposed an end-to-end deep learning framework i.e., IMO-TILs that can integrate pathological image with multi-omics data (i.e., mRNA and miRNA) to analyze TILs and explore the survival-associated interactions between TILs and tumors. Specifically, we firstly apply the graph attention network to describe the spatial interactions between TILs and tumor regions in WSIs. As to genomic data, the Concrete AutoEncoder (i.e., CAE) is adopted to select survival-associated Eigengenes from the high-dimensional multi-omics data. Finally, the deep generalized canonical correlation analysis (DGCCA) accompanied with the attention layer is implemented to fuse the image and multi-omics data for prognosis prediction of human cancers. The experimental results on three cancer cohorts derived from the Cancer Genome Atlas (TCGA) indicated that our method can both achieve higher prognosis results and identify consistent imaging and multi-omics bio-markers correlated strongly with the prognosis of human cancers.

Low-Dose Computed Tomography (LDCT) technique, which reduces the radiation harm to human bodies, is now attracting increasing interest in the medical imaging field. As the image quality is degraded by low dose radiation, LDCT exams require specialized reconstruction methods or denoising algorithms. However, most of the recent effective methods overlook the inner-structure of the original projection data (sinogram) which limits their denoising ability. The inner-structure of the sinogram represents special characteristics of the data in the sinogram domain. By maintaining this structure while denoising, the noise can be obviously restrained. Therefore, we propose an LDCT denoising network namely Sinogram Inner-Structure Transformer (SIST) to reduce the noise by utilizing the inner-structure in the sinogram domain. Specifically, we study the CT imaging mechanism and statistical characteristics of sinogram to design the sinogram inner-structure loss including the global and local inner-structure for restoring high-quality CT images. Besides, we propose a sinogram transformer module to better extract sinogram features. The transformer architecture using a self-attention mechanism can exploit interrelations between projections of different view angles, which achieves an outstanding performance in sinogram denoising. Furthermore, in order to improve the performance in the image domain, we propose the image reconstruction module to complementarily denoise both in the sinogram and image domain.

Multi-modal fusion has become an important data analysis technology in Alzheimer's disease (AD) diagnosis, which is committed to effectively extract and utilize complementary information among different modalities. However, most of the existing fusion methods focus on pursuing common feature representation by transformation, and ignore discriminative structural information among samples. In addition, most fusion methods use high-order feature extraction, such as deep neural network, by which it is difficult to identify biomarkers. In this paper, we propose a novel method named deep multi-modal discriminative and interpretability network (DMDIN), which aligns samples in a discriminative common space and provides a new approach to identify significant brain regions (ROIs) in AD diagnosis. Specifically, we reconstruct each modality with a hierarchical representation through multilayer perceptron (MLP), and take advantage of the shared self-expression coefficients constrained by diagonal blocks to embed the structural information of inter-class and the intra-class. Further, the generalized canonical correlation analysis (GCCA) is adopted as a correlation constraint to generate a discriminative common space, in which samples of the same category gather while samples of different categories stay away. Finally, in order to enhance the interpretability of the deep learning model, we utilize knowledge distillation to reproduce coordinated representations and capture influence of brain regions in AD classification. Experiments show that the proposed method performs better than several state-of-the-art methods in AD diagnosis.

Predicting fluid intelligence via neuroimaging data is important to understand neural mechanisms underlying diverse complex cognitive tasks in human brain. Functional connectivity (FC) reflects interactions among brain regions providing rich information of brain organization, which has been widely used in various behavior predictions. With the success of deep neural networks, graph convolutional network (GCN) is regarded as a promising feature learning method in FC networks (FCNs). However, as a challenging task, the existing GCN models cannot achieve a satisfactory performance in fluid intelligence predication due to the insufficient information utilization of brain connectivity and the limitation of graph convolution layer. To tackle these problems, this paper developed a Multi-Scale Multi-Hop GCN (MS-MH-GCN) to estimate fluid intelligence score by using FC. In the proposed method, we considered the hierarchy of brain system and thus utilized FCs from multiple spatial scales as input for the subsequent feature learning to achieve a complete characterize of brain organization for each individual. We also designed a new multi-hop graph convolution layer that uses multi-hop neighbors instead of l-hop neighbor in traditional GCN to guide message passing of nodal feature at every step. The introduction of high-order graph information benefits to the model learning ability improvement. Additionally, it is also worth emphasizing that, during feature learning process, we added contrast constraint to multi-scale FCNs to improve the similarity of feature representations across different spatial scales within a subject. Experimental results showed that our proposed method performed much better than the other four art-of-the-state methods.

In recent years, Electroencephalogram (EEG)-based emotion recognition has developed rapidly and gained increasing attention in the field of brain-computer interface. Relevant studies in the neuroscience domain have shown that various emotional states may activate differently in brain regions and time points. Though the EEG signals have the characteristics of high temporal resolution and strong global correlation, the low signal-to-noise ratio and much redundant information bring challenges to the fast emotion recognition. To cope with the above problem, we propose a Temporal and channel Transformer (TcT) model for emotion recognition, which is directly applied to the raw preprocessed EEG data. In the model, we propose a TcT self-attention mechanism that simultaneously captures temporal and channel dependencies. The sliding window weight sharing strategy is designed to gradually refine the features from coarse time granularity, and reduce the complexity of the attention calculation. The original signal is passed between layers through the residual structure to integrate the features of different layers. We conduct experiments on the DEAP database to verify the effectiveness of the proposed model. The results show that the model achieves better classification performance in less time and with fewer resources than state-of-the-art methods.

Sparse-View CT (SVCT), which provides low-dose and high-speed CT imaging, plays an important role in the medical imaging area. As the decrease of projection views, the reconstructed image suffers from severe artifacts. To this end, recent works utilize deep learning methods to improve the imaging quality of SVCT and achieve promising performances. However, these methods mainly focus on the network design and modeling but overlook the importance of choosing projection views. To address this issue, this paper proposes a Projection-view LeArning Network (PLANet), which can estimate the importance of different view angles through reconstruction network training and select the projection views for high-quality image restoration. Specifically, we generate synthesized sparse-view sinograms by subsampling projections from full-view sinograms based on a learnable distribution, which can be learned through reconstruction network training. Thus, important image views can be selected to acquire sparse-view projection in imaging equipment. Furthermore, effective data augmentations are provided by the online generation of sparse-view sinogram to improve the stability and performance of reconstruction networks. In short, our method can select the important projection views and learn high-performance reconstruction networks in one unified deep-learning framework. Comprehensive experiments show that the proposed method achieves promising results compared to state-of-the-art methods, and the ablation studies also show the superiority of our proposed PLANet in terms of effectiveness and robustness.

Ribonucleic acid (RNA) is a pivotal nucleic acid that plays a crucial role in regulating many biological activities. Recently, one study utilized a machine learning algorithm to automatically classify RNA structural events generated by a Mycobacterium smegmatis porin A nanopore trap. Although it can achieve desirable classification results, compared with deep learning (DL) methods, this classic machine learning requires domain knowledge to manually extract features, which is sophisticated, labor-intensive and time-consuming. Meanwhile, the generated original RNA structural events are not strictly equal in length, which is incompatible with the input requirements of DL models. To alleviate this issue, we propose a sequence-to-sequence (S2S) module that transforms the unequal length sequence (UELS) to the equal length sequence. Furthermore, to automatically extract features from the RNA structural events, we propose a sequence-to-sequence neural network based on DL. In addition, we add an attention mechanism to capture vital information for classification, such as dwell time and blockage amplitude. Through quantitative and qualitative analysis, the experimental results have achieved about a 2% performance increase (accuracy) compared to the previous method. The proposed method can also be applied to other nanopore platforms, such as the famous Oxford nanopore. It is worth noting that the proposed method is not only aimed at pursuing state-of-the-art performance but also provides an overall idea to process nanopore data with UELS.

MOTIVATION: As a rising research topic, brain imaging genetics aims to investigate the potential genetic architecture of both brain structure and function. It should be noted that in the brain, not all variations are deservedly caused by genetic effect, and it is generally unknown which imaging phenotypes are promising for genetic analysis. RESULTS: In this work, genetic variants (i.e. the single nucleotide polymorphism, SNP) can be correlated with brain networks (i.e. quantitative trait, QT), so that the connectome (including the brain regions and connectivity features) of functional brain networks from the functional magnetic resonance imaging data is identified. Specifically, a connection matrix is firstly constructed, whose upper triangle elements are selected to be connectivity features. Then, the PageRank algorithm is exploited for estimating the importance of different brain regions as the brain region features. Finally, a deep self-reconstruction sparse canonical correlation analysis (DS-SCCA) method is developed for the identification of genetic associations with functional connectivity phenotypic markers. This approach is a regularized, deep extension, scalable multi-SNP-multi-QT method, which is well-suited for applying imaging genetic association analysis to the Alzheimer's Disease Neuroimaging Initiative datasets. It is further optimized by adopting a parametric approach, augmented Lagrange and stochastic gradient descent. Extensive experiments are provided to validate that the DS-SCCA approach realizes strong associations and discovers functional connectivity and brain region phenotypic biomarkers to guide disease interpretation. AVAILABILITY AND IMPLEMENTATION: The Matlab code is available at https://github.com/meimeiling/DS-SCCA/tree/main. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Chronic obstructive pulmonary disease (COPD) is a prevalent chronic disease with high morbidity and mortality. The early diagnosis of COPD is vital for clinical treatment, which helps patients to have a better quality of life. Because COPD can be ascribed to chronic bronchitis and emphysema, lesions in a computed tomography (CT) image can present anywhere inside the lung with different types, shapes and sizes. Multiple instance learning (MIL) is an effective tool for solving COPD discrimination. In this study, a novel graph convolutional MIL with the adaptive additive margin loss (GCMIL-AAMS) approach is proposed to diagnose COPD by CT. Specifically, for those early stage patients, the selected instance-level features can be more discriminative if they were learned by our proposed graph convolution and pooling with self-attention mechanism. The AAMS loss can utilize the information of COPD severity on a hypersphere manifold by adaptively setting the angular margins to improve the performance, as the severity can be quantified as four grades by pulmonary function test. The results show that our proposed GCMIL-AAMS method provides superior discrimination and generalization abilities in COPD discrimination, with areas under a receiver operating characteristic curve (AUCs) of 0.960 ± 0.014 and 0.862 ± 0.010 in the test set and external testing set, respectively, in 5-fold stratified cross validation; moreover, it demonstrates that graph learning is applicable to MIL and suggests that MIL may be adaptable to graph learning.

Computer-aided early diagnosis of Alzheimer's disease (AD) and its prodromal form mild cognitive impairment (MCI) based on structure Magnetic Resonance Imaging (sMRI) has provided a cost-effective and objective way for early prevention and treatment of disease progression, leading to improved patient care. In this work, we have proposed a novel computer-aided approach for early diagnosis of AD by introducing an explainable 3D Residual Attention Deep Neural Network (3D ResAttNet) for end-to-end learning from sMRI scans. Different from the existing approaches, the novelty of our approach is three-fold: 1) A Residual Self-Attention Deep Neural Network has been proposed to capture local, global and spatial information of MR images to improve diagnostic performance; 2) An explainable method using Gradient-based Localization Class Activation mapping (Grad-CAM) has been introduced to improve the interpretability of the proposed method; 3) This work has provided a full end-to-end learning solution for automated disease diagnosis. Our proposed 3D ResAttNet method has been evaluated on a large cohort of subjects from real datasets for two changeling classification tasks (i.e. Alzheimer's disease (AD) vs. Normal cohort (NC) and progressive MCI (pMCI) vs. stable MCI (sMCI)). The experimental results show that the proposed approach has a competitive advantage over the state-of-the-art models in terms of accuracy performance and generalizability. The explainable mechanism in our approach is able to identify and highlight the contribution of the important brain parts (e.g., hippocampus, lateral ventricle and most parts of the cortex) for transparent decisions.

Multi-modal imaging data fusion has attracted much attention in medical data analysis because it can provide complementary information for more accurate analysis. Integrating functional and structural multi-modal imaging data has been increasingly used in the diagnosis of brain diseases, such as epilepsy. Most of the existing methods focus on the feature space fusion of different modalities but ignore the valuable high-order relationships among samples and the discriminative fused features for classification. In this paper, we propose a novel framework by fusing data from two modalities of functional MRI (fMRI) and diffusion tensor imaging (DTI) for epilepsy diagnosis, which effectively captures the complementary information and discriminative features from different modalities by high-order feature extraction with the attention mechanism. Specifically, we propose a triple network to explore the discriminative information from the high-order representation feature space learned from multi-modal data. Meanwhile, self-attention is introduced to adaptively estimate the degree of importance between brain regions, and the cross-attention mechanism is utilized to extract complementary information from fMRI and DTI. Finally, we use the triple loss function to adjust the distance between samples in the common representation space. We evaluate the proposed method on the epilepsy dataset collected from Jinling Hospital, and the experiment results demonstrate that our method is significantly superior to several state-of-the-art diagnosis approaches.

Deep neural networks are discovered to be non-robust when attacked by imperceptible adversarial examples, which is dangerous for it applied into medical diagnostic system that requires high reliability. However, the defense methods that have good effect in natural images may not be suitable for medical diagnostic tasks. The pre-processing methods (e.g., random resizing, compression) may lead to the loss of the small lesions feature in the medical image. Retraining the network on the augmented data set is also not practical for medical models that have already been deployed online. Accordingly, it is necessary to design an easy-to-deploy and effective defense framework for medical diagnostic tasks. In this paper, we propose a Robust and Retrain-Less Diagnostic Framework for Medical pretrained models against adversarial attack (i.e., MedRDF). It acts on the inference time of the pretrained medical model. Specifically, for each test image, MedRDF firstly creates a large number of noisy copies of it, and obtains the output labels of these copies from the pretrained medical diagnostic model. Then, based on the labels of these copies, MedRDF outputs the final robust diagnostic result by majority voting. In addition to the diagnostic result, MedRDF produces the Robust Metric (RM) as the confidence of the result. Therefore, it is convenient and reliable to utilize MedRDF to convert pretrained non-robust diagnostic models into robust ones. The experimental results on COVID-19 and DermaMNIST datasets verify the effectiveness of our MedRDF in improving the robustness of medical diagnostic models.

In recent years, numerous studies have adopted rs-fMRI to construct dynamic functional connectivity networks (DFCNs) and applied them to the diagnosis of brain diseases, such as epilepsy and schizophrenia. Compared with the static brain networks, the DFCNs have a natural advantage in reflecting the process of brain activity due to the time information contained in it. However, most of the current methods for constructing DFCNs fail to aggregate the brain topology structure and temporal variation of the functional architecture associated with brain regions, and often ignore the inherent multi-dimensional feature representation of DFCNs for classification. In order to address these issues, we propose a novel DFCNs construction and representation method and apply it to brain disease diagnosis. Specifically, we fuse the blood oxygen level dependent (BOLD) signal and interactions between brain regions to distinguish the brain topology within each time domain and across different time domains, by embedding block structure in the adjacency matrix. After that, a sparse tensor decomposition method with sparse local structure preserving regularization is developed to extract DFCNs features from a multi-dimensional perspective. Finally, the kernel discriminant analysis is employed to provide the decision result. We validate the proposed method on epilepsy and schizophrenia identification tasks, respectively. The experimental results show that the proposed method outperforms several state-of-the-art methods in the diagnosis of brain diseases.

Detecting the time-varying community structure of brain functional network is very important to reveal dynamic properties of the human brain. Although several community detection methods have been proposed, they are limited in real application due to their poor performance in large dynamic network and difficulty in parameter setting without prior knowledge. To address these problems, this paper proposes a novel dynamic community detection method for the brain network based on random walk, named as ML-RW. This method uses local community discovery instead of global community detection to improve its ability to deal with large dynamic networks. Specifically, ML-RW first selects a query brain region and sends out multiple random walkers starting from this region to explore local community structures of all networks in dynamic network simultaneously. It updates the visiting probability vector of each walker by aggregating the transition probabilities from itself and two temporally adjacent networks. Since the influence strength from one network to another is adaptively tuned according to the relevance of visiting histories of two networks, ML-RW could guarantee the temporal smoothness of the detected modular structure without introducing the hyper-parameter and thus avoids the problem of parameter setting in existing methods. Experiments on two public real neuroimaging datasets demonstrate that our proposed method has more potential to capture subtle community variations in the brain region, stronger ability to discover biomarkers for brain diseases, and higher test-retest reliability than the conventional method.

optimizing spectral filters for hyperspectral reconstruction has received increasing attentions recently. However, current filter selection methods suffer from extremely high computational complexity due to exhaustive optimization. In this paper, in order to reduce the computational complexity, we propose a novel Filter Selection Network (FS-Net) to select filters and learn the reconstruction network simultaneously. Specifically, we propose an end-to-end method to embed filter selection in FS-Net by setting spectral response functions as the input layer. Furthermore, we propose a non-negative Ll sparse regularization (NN-LI) to select optical filters automatically by sparsifying the input layer. Besides, we develop a two-stage training strategy for adjusting the number of selected filters. Experiments on public datasets show that our proposed method can considerably improve the reconstruction quality.

So far, in the study of neurological brain disorder diagnosis, there are mainly two kinds of work that exploits graph via Graph Convolutional Networks (GCN). Though they have achieved remarkable success, neither of them is able to simultaneously account for the brain region level correlation and subject level correlation. To tackle this issue, we design a Graph-In-Graph Convolutional Networks(GIGCN) framework, which turns out to inherit the merits of the two kinds of existing work. Specifically, we propose a graph-in-graph data structure that consists of two types of graph, namely internal brain connective graph and external population graph. In this data structure, an internal graph is assigned to each subject whose nodes represent brain regions and edges incorporate associations between brain regions, while the external graph is a population graph generated from the demographic of all subjects where edges describe the similarity between subjects. After that, our proposed framework employs GCN to explore some deeper relationships for better disease prediction. Results on the ABIDE dataset validate the effectiveness of our proposed method.

This paper presents the first study on Sign-aware Perturbations Regression (SaPR), where the observed response variables contain the aware sign (negative or positive) perturbations.In order to predict the non-perturbation response variables, we propose a novel parameter estimator SZOM (i.e.,Setting Zero Operator Method), which aims at taking full advantage of the aware perturbations information to correct the mistake values in the estimation process with computationally efficiency.In this paper, the two aspects of theoretical analysis are proposed to deeply understand our method.Firstly, we establish the perturbation parameter error upper bound and prove consistency guarantee in the linear regression scenario.Secondly, we introduce the generalization error bound for the proposed SZMO, which indicates that the error bound is related to the value and the number of negative and positive perturbations.The effectiveness of the proposed approach is well validated by the experimental results on both synthetic and real datasets.

Sparse canonical correlation analysis (SCCA) is a bi-multivariate technique used in imaging genetics to identify complex multi-SNP-multi-QT associations. However, the traditional SCCA algorithm has been designed to seek a linear correlation between the SNP genotype and brain imaging phenotype, ignoring the discriminant similarity information between within-class subjects in brain imaging genetics association analysis. In addition, multi-modality brain imaging phenotypes are extracted from different perspectives and imaging markers from the same region consistently showing up in multimodalities may provide more insights for the mechanistic understanding of diseases. In this paper, a novel multi-modality discriminant SCCA algorithm (MD-SCCA) is proposed to overcome these limitations as well as to improve learning results by incorporating valuable discriminant similarity information into the SCCA algorithm. Specifically, we first extract the discriminant similarity information between within-class subjects by the sparse representation. Second, the discriminant similarity information is enforced within SCCA to construct a discriminant SCCA algorithm (D-SCCA). At last, the MD-SCCA algorithm is adopted to fully explore the relationships among different modalities of different subjects. In experiments, both synthetic dataset and real data from the Alzheimer's Disease Neuroimaging Initiative database are used to test the performance of our algorithm. The empirical results have demonstrated that the proposed algorithm not only produces improved cross-validation performances but also identifies consistent cross-modality imaging genetic biomarkers.

The papers in this special section focus on multi-modal computing in biomedical applications. n recent years, the development of biomedical imaging techniques, integrative sensors, and machine learning, brings many benefits to the diagnosis of various diseases. We can collect, measure, and analyze vast volumes of health-related data using the technologies of computing and networking, leading to tremendous opportunities for the health and biomedical community. Meanwhile, these technologies have also brought new challenges and issues. Biomedical intelligence, especially precision medicine, is considered one of the most promising directions for healthcare development.

Contrast-enhanced ultrasound (CEUS) has emerged as a popular imaging modality in thyroid nodule diagnosis due to its ability to visualize vascular distribution in real time. Recently, a number of learning-based methods are dedicated to mine pathological-related enhancement dynamics and make prediction at one step, ignoring a native diagnostic dependency. In clinics, the differentiation of benign or malignant nodules always precedes the recognition of pathological types. In this paper, we propose a novel hierarchical temporal attention network (HiTAN) for thyroid nodule diagnosis using dynamic CEUS imaging, which unifies dynamic enhancement feature learning and hierarchical nodules classification into a deep framework. Specifically, this method decomposes the diagnosis of nodules into an ordered two-stage classification task, where diagnostic dependency is modeled by Gated Recurrent Units (GRUs). Besides, we design a local-to-global temporal aggregation (LGTA) operator to perform a comprehensive temporal fusion along the hierarchical prediction path. Particularly, local temporal information is defined as typical enhancement patterns identified with the guidance of perfusion representation learned from the differentiation level. Then, we leverage an attention mechanism to embed global enhancement dynamics into each identified salient pattern. In this study, we evaluate the proposed HiTAN method on the collected CEUS dataset of thyroid nodules. Extensive experimental results validate the efficacy of dynamic patterns learning, fusion and hierarchical diagnosis mechanism.

In the brain imaging genetic studies, it is a challenging task to estimate the association between quantitative traits (QTs) extracted from neuroimaging data and genetic markers such as single-nucleotide polymorphisms (SNPs). Most of the existing association studies are based on the extensions of sparse canonical correlation analysis (SCCA) for the identification of complex bi-multivariate associations, which can take the specific structure and group information into consideration. However, they often take the original data as input without considering its underlying complex multi-subspace structure, which will deteriorate the performance of the following integrative analysis. Accordingly, in this paper, the self-expressive property is exploited for the reconstruction of the original data before the association analysis, which can well describe the similarity structure. Specifically, we first apply the within-class similarity information to construct self-expressive networks by sparse representation. Then, we use the fusion method to iteratively fuse the self-expressive networks from multi-modality brain phenotypes into one network. Finally, we calculate the imaging genetic association based on the fused self-expressive network. We conduct the experiments on both single-modality and multi-modality phenotype data. Related experimental results validate that our method can not only better estimate the potential association between genetic markers and quantitative traits but also identify consistent multi-modality imaging genetic biomarkers to guide the interpretation of Alzheimer's disease.

Structural magnetic resonance imaging (sMRI) is widely used for the brain neurological disease diagnosis, which could reflect the variations of brain. However, due to the local brain atrophy, only a few regions in sMRI scans have obvious structural changes, which are highly correlative with pathological features. Hence, the key challenge of sMRI-based brain disease diagnosis is to enhance the identification of discriminative features. To address this issue, we propose a dual attention multi-instance deep learning network (DA-MIDL) for the early diagnosis of Alzheimer's disease (AD) and its prodromal stage mild cognitive impairment (MCI). Specifically, DA-MIDL consists of three primary components: 1) the Patch-Nets with spatial attention blocks for extracting discriminative features within each sMRI patch whilst enhancing the features of abnormally changed micro-structures in the cerebrum, 2) an attention multi-instance learning (MIL) pooling operation for balancing the relative contribution of each patch and yield a global different weighted representation for the whole brain structure, and 3) an attention-aware global classifier for further learning the integral features and making the AD-related classification decisions. Our proposed DA-MIDL model is evaluated on the baseline sMRI scans of 1689 subjects from two independent datasets (i.e., ADNI and AIBL). The experimental results show that our DA-MIDL model can identify discriminative pathological locations and achieve better classification performance in terms of accuracy and generalizability, compared with several state-of-the-art methods.

Aggregating multi-subject functional magnetic resonance imaging (fMRI) data is indispensable for generating valid and general inferences from patterns distributed across human brains. The disparities in anatomical structures and functional topographies of human brains warrant aligning fMRI data across subjects. However, the existing functional alignment methods cannot handle well various kinds of fMRI datasets today, especially when they are not temporally-aligned, i.e., some of the subjects probably lack the responses to some stimuli, or different subjects might follow different sequences of stimuli. In this paper, a cross-subject graph that depicts the (dis)similarities between samples across subjects is used as a priori for developing a more flexible framework that suits an assortment of fMRI datasets. However, the high dimension of fMRI data and the use of multiple subjects makes the crude framework time-consuming or unpractical. To address this issue, we further regularize the framework, so that a novel feasible kernel-based optimization, which permits non-linear feature extraction, could be theoretically developed. Specifically, a low-dimension assumption is imposed on each new feature space to avoid overfitting caused by the high-spatial-low-temporal resolution of fMRI data. Experimental results on five datasets suggest that the proposed method is not only superior to several state-of-the-art methods on temporally-aligned fMRI data, but also suitable for dealing with temporally-unaligned fMRI data.

In Compressive Sensing MRI (CS-MRI), measurement matrix learning has been developed as a promising method for measurement matrix designing. Research on MRI measurement task suggests that Relative 2-Norm Error (RLNE) of measurement images is imbalanced. However, current learning-based investigations suffer from the lack of probing imbalanced characteristic on measurement matrix learning. In this paper, we propose a novel Measurement Matrix Learning via Correlation Reweighting (MML-CR) approach for exploring and solving this problem by optimizing reweighted model.Specifically,we introduce a reweighting expected minimization model to obtain an essential measurement matrix in k-space. Besides, we propose an example correlation regularizer to prevent trivial solution for learning weights. Furthermore, we present an alternating solution and perform convergence analysis for the optimization. We also demonstrate quantitative and qualitative experimental results which show that our algorithm outperforms several state-of-art measurements methods. Compared with conventional methods, MML-CR achieves better performance on universal task.

Multi-voxel pattern analysis (MVPA) learns predictive models from task-based functional magnetic resonance imaging (fMRI) data, for distinguishing when subjects are performing different cognitive tasks — e.g., watching movies or making decisions. MVPA works best with a well-designed feature set and an adequate sample size. However, most fMRI datasets are noisy, high-dimensional, expensive to collect, and with small sample sizes. Further, training a robust, generalized predictive model that can analyze homogeneous cognitive tasks provided by multi-site fMRI datasets has additional challenges. This paper proposes the Shared Space Transfer Learning (SSTL) as a novel transfer learning (TL) approach that can functionally align homogeneous multi-site fMRI datasets, and so improve the prediction performance in every site. SSTL first extracts a set of common features for all subjects in each site. It then uses TL to map these site-specific features to a site-independent shared space in order to improve the performance of the MVPA. SSTL uses a scalable optimization procedure that works effectively for high-dimensional fMRI datasets. The optimization procedure extracts the common features for each site by using a single-iteration algorithm and maps these site-specific common features to the site-independent shared space. We evaluate the effectiveness of the proposed method for transferring between various cognitive tasks. Our comprehensive experiments validate that SSTL achieves superior performance to other state-of-the-art analysis techniques.

Segmenting bioimage based filaments is a critical step in a wide range of applications, including neuron reconstruction and blood vessel tracing. To achieve an acceptable segmentation performance, most of the existing methods need to annotate amounts of filamentary images in the training stage. Hence, these methods have to face the common challenge that the annotation cost is usually high. To address this problem, we propose an interactive segmentation method to actively select a few super-pixels for annotation, which can alleviate the burden of annotators. Specifically, we first apply a Simple Linear Iterative Clustering (i.e., SLIC) algorithm to segment filamentary images into compact and consistent super-pixels, and then propose a novel batch-mode based active learning method to select the most representative and informative (i.e., BMRI) super-pixels for pixel-level annotation. We then use a bagging strategy to extract several sets of pixels from the annotated super-pixels, and further use them to build different Laplacian Regularized Gaussian Mixture Models (Lap-GMM) for pixel-level segmentation. Finally, we perform the classifier ensemble by combining multiple Lap-GMM models based on a majority voting strategy. We evaluate our method on three public available filamentary image datasets. Experimental results show that, to achieve comparable performance with the existing methods, the proposed algorithm can save 40 percent annotation efforts for experts.

Multi-atlas based segmentation methods have shown their effectiveness in brain regions-of-interesting (ROIs) segmentation, by propagating labels from multiple atlases to a target image based on the similarity between patches in the target image and multiple atlas images. Most of the existing multiatlas based methods use image intensity features to calculate the similarity between a pair of image patches for label fusion. In particular, using only low-level image intensity features cannot adequately characterize the complex appearance patterns (e.g., the high-order relationship between voxels within a patch) of brain magnetic resonance (MR) images. To address this issue, this paper develops a high-order feature learning framework for multi-atlas based label fusion, where high-order features of image patches are extracted and fused for segmenting ROIs of structural brain MR images. Specifically, an unsupervised feature learning method (i.e., means-covariances restricted Boltzmann machine, mcRBM) is employed to learn high-order features (i.e., mean and covariance features) of patches in brain MR images. Then, a group-fused sparsity dictionary learning method is proposed to jointly calculate the voting weights for label fusion, based on the learned high-order and the original image intensity features. The proposed method is compared with several state-of-the-art label fusion methods on ADNI, NIREP and LONI-LPBA40 datasets. The Dice ratio achieved by our method is 88:30%, 88:83%, 79:54% and 81:02% on left and right hippocampus on the ADNI, NIREP and LONI-LPBA40 datasets, respectively, while the best Dice ratio yielded by the other methods are 86:51%, 87:39%, 78:48% and 79:65% on three datasets, respectively.

Chest computed tomography (CT) becomes an effective tool to assist the diagnosis of coronavirus disease-19 (COVID-19). Due to the outbreak of COVID-19 worldwide, using the computed-aided diagnosis technique for COVID-19 classification based on CT images could largely alleviate the burden of clinicians. In this paper, we propose an Adaptive Feature Selection guided Deep Forest (AFS-DF) for COVID-19 classification based on chest CT images. Specifically, we first extract location-specific features from CT images. Then, in order to capture the high-level representation of these features with the relatively small-scale data, we leverage a deep forest model to learn high-level representation of the features. Moreover, we propose a feature selection method based on the trained deep forest model to reduce the redundancy of features, where the feature selection could be adaptively incorporated with the COVID-19 classification model. We evaluated our proposed AFS-DF on COVID-19 dataset with 1495 patients of COVID-19 and 1027 patients of community acquired pneumonia (CAP). The accuracy (ACC), sensitivity (SEN), specificity (SPE), AUC, precision and F1-score achieved by our method are 91.79%, 93.05%, 89.95%, 96.35%, 93.10% and 93.07%, respectively. Experimental results on the COVID-19 dataset suggest that the proposed AFS-DF achieves superior performance in COVID-19 vs. CAP classification, compared with 4 widely used machine learning methods.

OBJECTIVE: Two-dimensional fluoroscopy is the standard guidance imaging method for closed endovascular intervention. However, two-dimensional fluoroscopy lacks depth perception for the intervention catheter and causes radiation exposure for both surgeons and patients. In this paper, we extend our previous study and develop the improved three-dimensional (3D) catheter shape estimation using ultrasound imaging. In addition, we perform further quantitative evaluations of endovascular navigation. METHOD: First, the catheter tracking accuracy in ultrasound images is improved by adjusting the state vector and adding direction information. Then, the 3D catheter points from the catheter tracking are further optimized based on the 3D catheter shape optimization with a high-quality sample set. Finally, the estimated 3D catheter shapes from ultrasound images are overlaid with preoperative 3D tissue structures for the intuitive endovascular navigation. RESULTS: the tracking accuracy of the catheter increased by 24.39%, and the accuracy of the catheter shape optimization step also increased by approximately 17.34% compared with our previous study. Furthermore, the overall error of catheter shape estimation was further validated in the catheter intervention experiment of in vitro cardiovascular tissue and in a vivo swine, and the errors were 2.13 mm and 3.37 mm, respectively. CONCLUSION: Experimental results demonstrate that the improved catheter shape estimation using ultrasound imaging is accurate and appropriate for endovascular navigation. SIGNIFICANCE: Improved navigation reduces the radiation risk because it decreases use of X-ray imaging. In addition, this navigation method can also provide accurate 3D catheter shape information for endovascular surgery.

Currently, how to conjointly fuse structural connectivity (SC) and functional connectivity (FC) for identifying brain diseases is a hot topic in the area of brain network analysis. Most of the existing works combine two types of connectivity in decision level, thus ignoring the underlying relationship between SC and FC. To solve this problem, in this paper, we model the brain network as the multi-layer network formed by the SC and FC, and then propose a coherent pattern to represent structural information of the multi-layer network for the brain disease identification. The proposed coherent pattern consists of a paired-subgraph extracted from the FC and SC within the same node-set. Compared with the previous methods, this coherent pattern not only describes the connectivity information of both SC and FC by subgraphs at each layer, but also reflects their intrinsic relationship by the co-occurrence pattern of the paired-subgraph. Based on this coherent pattern, we further develop a framework for identifying brain diseases. Specifically, we first construct multi-layer networks by using SC and FC for each subject and then mine coherent patterns that frequently appear in each group. Next, we select the discriminative coherent pattern from these frequent coherent patterns according to their frequency of occurrence. Finally, we construct a feature matrix for each subject based on the binary indicator vector and then use the support vector machine (SVM) as its classifier. Experimental results on real epilepsy datasets demonstrate that our method outperforms several state-of-the-art approaches in the tasks of brain disease classification.

Brain network provides essential insights in diagnosing many brain disorders. Integrative analysis of multiple types of connectivity, e.g, functional connectivity (FC) and structural connectivity (SC), can take advantage of their complementary information and therefore may help to identify patients. However, traditional brain network methods usually focus on either FC or SC for describing node interactions and only consider the interaction between paired network nodes. To tackle this problem, in this paper, we propose an Attention-Diffusion-Bilinear Neural Network (ADB-NN) framework for brain network analysis, which is trained in an end-to-end manner. The proposed network seamlessly couples FC and SC to learn wider node interactions and generates a joint representation of FC and SC for diagnosis. Specifically, a brain network (graph) is first defined, where each node corresponding to a brain region is governed by the features of brain activities (i.e., FC) extracted from functional magnetic resonance imaging (fMRI), and the presence of edges is determined by neural fiber physical connections (i.e., SC) extracted from Diffusion Tensor Imaging (DTI). Based on this graph, we train two Attention-Diffusion-Bilinear (ADB) modules jointly. In each module, an attention model is utilized to automatically learn the strength of node interactions. This information further guides a diffusion process that generates new node representations by considering the influence from other nodes as well. After that, the second-order statistics of these node representations are extracted by bilinear pooling to form connectivity-based features for disease prediction. The two ADB modules correspond to the one-step and two-step diffusion, respectively. Experiments on a real epilepsy dataset demonstrate the effectiveness and advantages of our proposed method.

The integrative analysis of histopathological images and genomic data has received increasing attention for studying the complex mechanisms of driving cancers. However, most image-genomic studies have been restricted to combining histopathological images with the single modality of genomic data (e.g., mRNA transcription or genetic mutation), and thus neglect the fact that the molecular architecture of cancer is manifested at multiple levels, including genetic, epigenetic, transcriptional, and post-transcriptional events. To address this issue, we propose a novel ordinal multi-modal feature selection (OMMFS) framework that can simultaneously identify important features from both pathological images and multi-modal genomic data (i.e., mRNA transcription, copy number variation, and DNA methylation data) for the prognosis of cancer patients. Our model is based on a generalized sparse canonical correlation analysis framework, by which we also take advantage of the ordinal survival information among different patients for survival outcome prediction. We evaluate our method on three early-stage cancer datasets derived from The Cancer Genome Atlas (TCGA) project, and the experimental results demonstrated that both the selected image and multi-modal genomic markers are strongly correlated with survival enabling effective stratification of patients with distinct survival than the comparing methods, which is often difficult for early-stage cancer patients.

Brain region-of-interest (ROI) segmentation based on structural magnetic resonance imaging (MRI) scans is an essential step for many computer-aid medical image analysis applications. Due to low intensity contrast around ROI boundary and large inter-subject variance, it has been remaining a challenging task to effectively segment brain ROIs from structural MR images. Even though several deep learning methods for brain MR image segmentation have been developed, most of them do not incorporate shape priors to take advantage of the regularity of brain structures, thus leading to sub-optimal performance. To address this issue, we propose an anatomical attention guided deep learning framework for brain ROI segmentation of structural MR images, containing two subnetworks. The first one is a segmentation subnetwork, used to simultaneously extract discriminative image representation and segment ROIs for each input MR image. The second one is an anatomical attention subnetwork, designed to capture the anatomical structure information of the brain from a set of labeled atlases. To utilize the anatomical attention knowledge learned from atlases, we develop an anatomical gate architecture to fuse feature maps derived from a set of atlas label maps and those from the to-be-segmented image for brain ROI segmentation. In this way, the anatomical prior learned from atlases can be explicitly employed to guide the segmentation process for performance improvement. Within this framework, we develop two anatomical attention guided segmentation models, denoted as anatomical gated fully convolutional network (AG-FCN) and anatomical gated U-Net (AG-UNet), respectively. Experimental results on both ADNI and LONI-LPBA40 datasets suggest that the proposed AG-FCN and AG-UNet methods achieve superior performance in ROI segmentation of brain MR images, compared with several state-of-the-art methods.

Autism spectrum disorder (ASD) is a neurodevelopmental disorder that is characterized by a wide range of symptoms. Identifying biomarkers for accurate diagnosis is crucial for early intervention of ASD. While multi-site data increase sample size and statistical power, they suffer from inter-site heterogeneity. To address this issue, we propose a multi-site adaption framework via low-rank representation decomposition (maLRR) for ASD identification based on functional MRI (fMRI). The main idea is to determine a common low-rank representation for data from the multiple sites, aiming to reduce differences in data distributions. Treating one site as a target domain and the remaining sites as source domains, data from these domains are transformed (i.e., adapted) to a common space using low-rank representation. To reduce data heterogeneity between the target and source domains, data from the source domains are linearly represented in the common space by those from the target domain. We evaluated the proposed method on both synthetic and real multi-site fMRI data for ASD identification. The results suggest that our method yields superior performance over several state-of-the-art domain adaptation methods.

Generative Adversarial Networks (GANs) are powerful tools for reconstructing Compressed Sensing Magnetic Resonance Imaging (CS-MRI). However most recent works lack exploration of structure information of MRI images that is crucial for clinical diagnosis. To tackle this problem, we propose the Structure-Enhanced GAN (SEGAN) that aims at restoring structure information at both local and global scale. SEGAN defines a new structure regularization called Patch Correlation Regularization (PCR) which allows for efficient extraction of structure information. In addition, to further enhance the ability to uncover structure information, we propose a novel generator SU-Net by incorporating multiple-scale convolution filters into each layer. Besides, we theoretically analyze the convergence of stochastic factors contained in training process. Experimental results show that SEGAN is able to learn target structure information and achieves state-of-theart performance for CS-MRI reconstruction.

Brain network analysis can help reveal the pathological basis of neurological disorders and facilitate automated diagnosis of brain diseases, by exploring connectivity patterns in the human brain. Effectively representing the brain network has always been the fundamental task of computeraided brain network analysis. Previous studies typically utilize human-engineered features to represent brain connectivity networks, but these features may not be well coordinated with subsequent classifiers. Besides, brain networks are often equipped with multiple hubs (i.e., nodes occupying a central position in the overall organization of a network), providing essential clues to describe connectivity patterns. However, existing studies often fail to explore such hubs from brain connectivity networks. To address these two issues, we propose a Connectivity Network analysis method with discriminative Hub Detection (CNHD) for brain disease diagnosis using functional magnetic resonance imaging (fMRI) data. Specifically, we incorporate both feature extraction of brain networks and network-based classification into a unified model, while discriminative hubs can be automatically identified from data via ℓ1-norm and ℓ2,1-norm regularizers. The proposed CNHD method is evaluated on three real-world schizophrenia datasets with fMRI scans. Experimental results demonstrate that our method not only outperforms several state-of-the-art approaches in disease diagnosis, but also is effective in automatically identifying disease-related network hubs in the human brain.

MOTIVATION: Neuroimaging genetics is an emerging field to identify the associations between genetic variants [e.g. single-nucleotide polymorphisms (SNPs)] and quantitative traits (QTs) such as brain imaging phenotypes. However, most of the current studies focus only on the associations between brain structure imaging and genetic variants, while neglecting the connectivity information between brain regions. In addition, the brain itself is a complex network, and the higher-order interaction may contain useful information for the mechanistic understanding of diseases [i.e. Alzheimer's disease (AD)]. RESULTS: A general framework is proposed to exploit network voxel information and network connectivity information as intermediate traits that bridge genetic risk factors and disease status. Specifically, we first use the sparse representation (SR) model to build hyper-network to express the connectivity features of the brain. The network voxel node features and network connectivity edge features are extracted from the structural magnetic resonance imaging (sMRI) and resting-state functional magnetic resonance imaging (fMRI), respectively. Second, a diagnosis-aligned multi-modality regression method is adopted to fully explore the relationships among modalities of different subjects, which can help further mine the relation between the risk genetics and brain network features. In experiments, all methods are tested on the Alzheimer's Disease Neuroimaging Initiative (ADNI) database. The experimental results not only verify the effectiveness of our proposed framework but also discover some brain regions and connectivity features that are highly related to diseases. AVAILABILITY AND IMPLEMENTATION: The Matlab code is available at http://ibrain.nuaa.edu.cn/2018/list.htm.

Summary Tongue image segmentation is a crucial step in developing an automatic tongue diagnosis system. After exploring characteristics of image thresholding in different color spaces, we propose a simple and effective tongue image segmentation method. Specifically, the proposed method first extracts an initial tongue body region by performing image thresholding on the transformed hue component in HSI (Hue, Saturation, and Intensity) color space. Then, one of two image thresholding results on the red component of an original tongue image is adaptively selected to find the gap region between the tongue body root and the upper lip. Finally, the initial tongue body region is refined by removing fake object regions such as the upper lip to obtain final tongue image segmentation result. Experimental results on a tongue image dataset containing 200 subjects demonstrate the effectiveness of the proposed method.

Imaging genetics has attracted significant interests in recent studies. Traditional work has focused on mass-univariate statistical approaches that identify important single nucleotide polymorphisms (SNPs) associated with quantitative traits (QTs) of brain structure or function. More recently, to address the problem of multiple comparison and weak detection, multivariate analysis methods such as the least absolute shrinkage and selection operator (Lasso) are often used to select the most relevant SNPs associated with QTs. However, one problem of Lasso, as well as many other feature selection methods for imaging genetics, is that some useful prior information, e.g., the hierarchical structure among SNPs, are rarely used for designing a more powerful model. In this paper, we propose to identify the associations between candidate genetic features (i.e., SNPs) and magnetic resonance imaging (MRI)-derived measures using a tree-guided sparse learning (TGSL) method. The advantage of our method is that it explicitly models the complex hierarchical structure among the SNPs in the objective function for feature selection. Specifically, motivated by the biological knowledge, the hierarchical structures involving gene groups and linkage disequilibrium (LD) blocks as well as individual SNPs are imposed as a tree-guided regularization term in our TGSL model. Experimental studies on simulation data and the Alzheimer's Disease Neuroimaging Initiative (ADNI) data show that our method not only achieves better predictions than competing methods on the MRI-derived measures of AD-related region of interests (ROIs) (i.e., hippocampus, parahippocampal gyrus, and precuneus), but also identifies sparse SNP patterns at the block level to better guide the biological interpretation.

The fractional amplitude of low-frequency fluctuations (fALFF) has been widely used as potential clinical biomarkers for resting-state functional-magnetic-resonance-imaging-based schizophrenia diagnosis. How-ever, previous studies usually measure the fALFF with specific bands from 0.01 to 0.08 Hz, which cannot fully delineate the complex variations of spontaneous fluctuations in the resting-state brain. In addition, fALFF data are intrinsically constrained by the brain structure, but most of the traditional methods have not consider it in feature selection. For addressing these problems, we propose a model to classify schizophrenia in multifrequency bands with tree-guided group sparse learning. In detail, we first acquire the fALFF data in multifrequency bands (i.e., slow-5: 0.01-0.027 Hz, slow-4: 0.027-0.073 Hz, slow-3: 0.073-0.198 Hz, and slow-2: 0.198-0.25 Hz). Then, we divide the whole brain into different candidate patches and select those significant patches related to schizophrenia using random forest-based important score. Moreover, we use tree-structured sparse learning method for feature selection with the above patch spatial constraint. Finally, considering biomarkers from multifrequency bands can reflect complementary information among multiple-frequency bands, we adopt the multikernel learning method to combine features of multifrequency bands for classification. Our experimental results show that these biomarkers from multifrequency bands can achieve a classification accuracy of 91.1% on 17 schizophrenia patients and 17 healthy controls, further demonstrating that the multifrequency bands analysis can better account for classification of schizophrenia.

Stock trading is a popular investment approach in real world. However, since lacking enough domain knowledge and experience, it is very difficult for common investors to analyze thousands of stocks manually. Algorithmic investment provides another rational way to formulate human knowledge as a trading agent. However, it still requires well-built knowledge and experience to design effective trading algorithms in such a volatile market. Fortunately, various kinds of historical trading records are easy to obtain in this big-data era, it is invaluable of us to extract the trading knowledge hidden in the data to help people make better decisions. In this paper, we propose a reinforcement learning driven Investor-Imitator framework to formalize the trading knowledge, by imitating an investor's behavior with a set of logic descriptors. In particular, to instantiate specific logic descriptors, we introduce the Rank-Invest model that can keep the diversity of logic descriptors by learning to optimize different evaluation metrics. In the experiment, we first simulate three types of investors, representing different degrees of information disclosure we may meet in real market. By learning towards these investors, we can tell the inherent trading logic of the target investor with the Investor-Imitator empirically, and the extracted interpretable knowledge can help us better understand and construct trading portfolios. Experimental results in this paper sufficiently demonstrate the designed purpose of Investor-Imitator, it makes the Investor-Imitator an applicable and meaningful intelligent trading framework in financial investment research.

Nowadays, with the advances in microscopic imaging, accurate classification of bioimage-based protein subcellular location pattern has attracted as much attention as ever. One of the basic challenging problems is how to select the useful feature components among thousands of potential features to describe the images. This is not an easy task especially considering there is a high ratio of multi-location proteins. Existing feature selection methods seldom take the correlation among different cellular compartments into consideration, and thus may miss some features that will be co-important for several subcellular locations. To deal with this problem, we make use of the important structural correlation among different cellular compartments and propose an organelle structural correlation regularized feature selection method CSF (Common-Sets of Features) in this paper. We formulate the multi-label classification problem by adopting a group-sparsity regularizer to select common subsets of relevant features from different cellular compartments. In addition, we also add a cell structural correlation regularized Laplacian term, which utilizes the prior biological structural information to capture the intrinsic dependency among different cellular compartments. The CSF provides a new feature selection strategy for multi-label bio-image subcellular pattern classifications, and the experimental results also show its superiority when comparing with several existing algorithms.

The wisdom of crowds (WOCs), as a theory in the social science, gets a new paradigm in computer science. The WOC theory explains that the aggregate decision made by a group is often better than those of its individual members if specific conditions are satisfied. This paper presents a novel framework for unsupervised and semisupervised cluster ensemble by exploiting the WOC theory. We employ four conditions in the WOC theory, i.e., diversity, independency, decentralization, and aggregation, to guide both constructing of individual clustering results and final combination for clustering ensemble. First, independency criterion, as a novel mapping system on the raw data set, removes the correlation between features on our proposed method. Then, decentralization as a novel mechanism generates high quality individual clustering results. Next, uniformity as a new diversity metric evaluates the generated clustering results. Further, weighted evidence accumulation clustering method is proposed for the final aggregation without using thresholding procedure. Experimental study on varied data sets demonstrates that the proposed approach achieves superior performance to state-of-the-art methods.

As a simple representation of interactions among distributed brain regions, brain networks have been widely applied to automated diagnosis of brain diseases, such as Alzheimer's disease (AD) and its early stage, i.e., mild cognitive impairment (MCI). In brain network analysis, a challenging task is how to measure the similarity between a pair of networks. Although many graph kernels (i.e., kernels defined on graphs) have been proposed for measuring the topological similarity of a pair of brain networks, most of them are defined using general graphs, thus ignoring the uniqueness of each node in brain networks. That is, each node in a brain network denotes a particular brain region, which is a specific characteristics of brain networks. Accordingly, in this paper, we construct a novel sub-network kernel for measuring the similarity between a pair of brain networks and then apply it to brain disease classification. Different from current graph kernels, our proposed sub-network kernel not only takes into account the inherent characteristic of brain networks, but also captures multi-level (from local to global) topological properties of nodes in brain networks, which are essential for defining the similarity measure of brain networks. To validate the efficacy of our method, we perform extensive experiments on subjects with baseline functional magnetic resonance imaging data obtained from the Alzheimer's disease neuroimaging initiative database. Experimental results demonstrate that the proposed method outperforms several state-of-the-art graph-based methods in MCI classification.

Brain connectivity networks based on magnetic resonance imaging (MRI) or functional MRI (fMRI) data provide a straightforward way to quantify the structural or functional systems of the brain. Currently, there are several network descriptors developed for representing and analyzing brain connectivity networks. However, most of them are designed for unweighted networks, regardless of the valuable weight information of edges, or do not take advantage of the ordinal relationship of weighted edges (even though they are designed for weighted networks). In this paper, we propose a new network descriptor (i.e., ordinal pattern that contains a sequence of weighted edges) for brain connectivity network analysis. Compared with previous network properties, the proposed ordinal patterns cannot only take advantage of the weight information of edges but also explicitly model the ordinal relationship of weighted edges in brain connectivity networks. We further develop an ordinal pattern-based learning framework for brain disease diagnosis using resting-state fMRI data. Specifically, we first construct a set of brain functional connectivity networks, where each network is corresponding to a particular subject. We then develop an algorithm to identify ordinal patterns that frequently appear in brain connectivity networks of patients and normal controls. We further perform discriminative ordinal pattern selection and extract feature representations for subjects based on the selected ordinal patterns, followed by a learning model for automated brain disease diagnosis. Experimental results on both Alzheimer's Disease Neuroimaging Initiative and attention deficit hyperactivity disorder-200 data sets demonstrate that our method outperforms the several state-of-the-art approaches in the tasks of disease classification and clinical score regression.

Multivariate Pattern (MVP) classification can map different cognitive states to the brain tasks. One of the main challenges in MVP analysis is validating the generated results across subjects. However, analyzing multi-subject fMRI data requires accurate functional alignments between neuronal activities of different subjects, which can rapidly increase the performance and robustness of the final results. Hyperalignment (HA) is one of the most effective functional alignment methods, which can be mathematically formulated by the Canonical Correlation Analysis (CCA) methods. Since HA mostly uses the unsupervised CCA techniques, its solution may not be optimized for MVP analysis. By incorporating the idea of Local Discriminant Analysis (LDA) into CCA, this paper proposes Local Discriminant Hyperalignment (LDHA) as a novel supervised HA method, which can provide better functional alignment for MVP analysis. Indeed, the locality is defined based on the stimuli categories in the train-set, where the correlation between all stimuli in the same category will be maximized and the correlation between distinct categories of stimuli approaches to near zero. Experimental studies on multi-subject MVP analysis confirm that the LDHA method achieves superior performance to other state-of-the-art HA algorithms.

This paper proposes Deep Hyperalignment (DHA) as a regularized, deep extension, scalable Hyperalignment (HA) method, which is well-suited for applying functional alignment to fMRI datasets with nonlinearity, high-dimensionality (broad ROI), and a large number of subjects. Unlink previous methods, DHA is not limited by a restricted fixed kernel function. Further, it uses a parametric approach, rank-m Singular Value Decomposition (SVD), and stochastic gradient descent for optimization. Therefore, DHA has a suitable time complexity for large datasets, and DHA does not require the training data when it computes the functional alignment for a new subject. Experimental studies on multi-subject fMRI analysis confirm that the DHA method achieves superior performance to other state-of-the-art HA algorithms.

Linear classifier is an essential part of machine learning, and improving its robustness has attracted much effort. Logistic regression (LR) is one of the most widely used linear classifier for its simplicity and probabilistic output. To reduce the risk of overfitting, LR was enhanced by introducing a generalized logistic loss (GLL) with a L2-norm regularization, aiming to maximize the minimum margin. However, the strategy of maximizing minimal margin is less robust to noisy data. In this paper, we incorporate GLL with margin distribution to exploit the statistical information from the training data, and propose a margin distribution logistic machine (MDLM) for better generalization performance and robustness. Furthermore, we extend MDLM to a multi-class version and learn different classes simultaneously by utilizing more information shared across these classes. Extensive experimental results validate the effectiveness of MDLM on both binary classification and multi-class classification.

Multivariate Pattern (MVP) classification holds enormous potential for decoding visual stimuli in the human brain by employing task-based fMRI data sets. There is a wide range of challenges in the MVP techniques, i.e. decreasing noise and sparsity, defining effective regions of interest (ROIs), visualizing results, and the cost of brain studies. In overcoming these challenges, this paper proposes a novel model of neural representation, which can automatically detect the active regions for each visual stimulus and then utilize these anatomical regions for visualizing and analyzing the functional activities. Therefore, this model provides an opportunity for neuroscientists to ask this question: what is the effect of a stimulus on each of the detected regions instead of just study the fluctuation of voxels in the manually selected ROIs. Moreover, our method introduces analyzing snapshots of brain image for decreasing sparsity rather than using the whole of fMRI time series. Further, a new Gaussian smoothing method is proposed for removing noise of voxels in the level of ROIs. The proposed method enables us to combine different fMRI data sets for reducing the cost of brain studies. Experimental studies on 4 visual categories (words, consonants, objects and nonsense photos) confirm that the proposed method achieves superior performance to state-of-the-art methods.

MOTIVATION: Neuroimaging genetics identifies the relationships between genetic variants (i.e., the single nucleotide polymorphisms) and brain imaging data to reveal the associations from genotypes to phenotypes. So far, most existing machine-learning approaches are widely used to detect the effective associations between genetic variants and brain imaging data at one time-point. However, those associations are based on static phenotypes and ignore the temporal dynamics of the phenotypical changes. The phenotypes across multiple time-points may exhibit temporal patterns that can be used to facilitate the understanding of the degenerative process. In this article, we propose a novel temporally constrained group sparse canonical correlation analysis (TGSCCA) framework to identify genetic associations with longitudinal phenotypic markers. RESULTS: The proposed TGSCCA method is able to capture the temporal changes in brain from longitudinal phenotypes by incorporating the fused penalty, which requires that the differences between two consecutive canonical weight vectors from adjacent time-points should be small. A new efficient optimization algorithm is designed to solve the objective function. Furthermore, we demonstrate the effectiveness of our algorithm on both synthetic and real data (i.e., the Alzheimer's Disease Neuroimaging Initiative cohort, including progressive mild cognitive impairment, stable MCI and Normal Control participants). In comparison with conventional SCCA, our proposed method can achieve strong associations and discover phenotypic biomarkers across multiple time-points to guide disease-progressive interpretation. AVAILABILITY AND IMPLEMENTATION: The Matlab code is available at https://sourceforge.net/projects/ibrain-cn/files/ . CONTACT: [email protected] or [email protected].

MOTIVATION: The systematic study of subcellular location pattern is very important for fully characterizing the human proteome. Nowadays, with the great advances in automated microscopic imaging, accurate bioimage-based classification methods to predict protein subcellular locations are highly desired. All existing models were constructed on the independent parallel hypothesis, where the cellular component classes are positioned independently in a multi-class classification engine. The important structural information of cellular compartments is missed. To deal with this problem for developing more accurate models, we proposed a novel cell structure-driven classifier construction approach (SC-PSorter) by employing the prior biological structural information in the learning model. Specifically, the structural relationship among the cellular components is reflected by a new codeword matrix under the error correcting output coding framework. Then, we construct multiple SC-PSorter-based classifiers corresponding to the columns of the error correcting output coding codeword matrix using a multi-kernel support vector machine classification approach. Finally, we perform the classifier ensemble by combining those multiple SC-PSorter-based classifiers via majority voting. RESULTS: We evaluate our method on a collection of 1636 immunohistochemistry images from the Human Protein Atlas database. The experimental results show that our method achieves an overall accuracy of 89.0%, which is 6.4% higher than the state-of-the-art method. AVAILABILITY AND IMPLEMENTATION: The dataset and code can be downloaded from https://github.com/shaoweinuaa/. CONTACT: [email protected] SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Error-correcting output coding (ECOC) is one of the most widely used strategies for dealing with multi-class problems by decomposing the original multi-class problem into a series of binary sub-problems. In traditional ECOC-based methods, binary classifiers corresponding to those sub-problems are usually trained separately without considering the relationships among these classifiers. However, as these classifiers are established on the same training data, there may be some inherent relationships among them. Exploiting such relationships can potentially improve the generalization performances of individual classifiers, and, thus, boost ECOC learning algorithms. In this paper, we explore to mine and utilize such relationship through a joint classifier learning method, by integrating the training of binary classifiers and the learning of the relationship among them into a unified objective function. We also develop an efficient alternating optimization algorithm to solve the objective function. To evaluate the proposed method, we perform a series of experiments on eleven datasets from the UCI machine learning repository as well as two datasets from real-world image recognition tasks. The experimental results demonstrate the efficacy of the proposed method, compared with state-of-the-art methods for ECOC-based multi-class classification.

Feature selection aims to identify the most informative features for a compact and accurate data representation. As typical supervised feature selection methods, Lasso and its variants using L1-norm-based regularization terms have received much attention in recent studies, most of which use class labels as supervised information. Besides class labels, there are other types of supervised information, e.g., pairwise constraints that specify whether a pair of data samples belong to the same class (must-link constraint) or different classes (cannot-link constraint). However, most of existing L1-norm-based sparse learning methods do not take advantage of the pairwise constraints that provide us weak and more general supervised information. For addressing that problem, we propose a pairwise constraint-guided sparse (CGS) learning method for feature selection, where the must-link and the cannot-link constraints are used as discriminative regularization terms that directly concentrate on the local discriminative structure of data. Furthermore, we develop two variants of CGS, including: 1) semi-supervised CGS that utilizes labeled data, pairwise constraints, and unlabeled data and 2) ensemble CGS that uses the ensemble of pairwise constraint sets. We conduct a series of experiments on a number of data sets from University of California-Irvine machine learning repository, a gene expression data set, two real-world neuroimaging-based classification tasks, and two large-scale attribute classification tasks. Experimental results demonstrate the efficacy of our proposed methods, compared with several established feature selection methods.

As shown in the literature, methods based on multiple templates usually achieve better performance, compared with those using only a single template for processing medical images. However, most existing multi-template based methods simply average or concatenate multiple sets of features extracted from different templates, which potentially ignores important structural information contained in the multi-template data. Accordingly, in this paper, we propose a novel relationship induced multi-template learning method for automatic diagnosis of Alzheimer's disease (AD) and its prodromal stage, i.e., mild cognitive impairment (MCI), by explicitly modeling structural information in the multi-template data. Specifically, we first nonlinearly register each brain's magnetic resonance (MR) image separately onto multiple pre-selected templates, and then extract multiple sets of features for this MR image. Next, we develop a novel feature selection algorithm by introducing two regularization terms to model the relationships among templates and among individual subjects. Using these selected features corresponding to multiple templates, we then construct multiple support vector machine (SVM) classifiers. Finally, an ensemble classification is used to combine outputs of all SVM classifiers, for achieving the final result. We evaluate our proposed method on 459 subjects from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database, including 97 AD patients, 128 normal controls (NC), 117 progressive MCI (pMCI) patients, and 117 stable MCI (sMCI) patients. The experimental results demonstrate promising classification performance, compared with several state-of-the-art methods for multi-template based AD/MCI classification.

Clustering explores meaningful patterns in the non-labeled data sets. Cluster Ensemble Selection (CES) is a new approach, which can combine individual clustering results for increasing the performance of the final results. Although CES can achieve better final results in comparison with individual clustering algorithms and cluster ensemble methods, its performance can be dramatically affected by its consensus diversity metric and thresholding procedure. There are two problems in CES: 1) most of the diversity metrics is based on heuristic Shannon's entropy and 2) estimating threshold values are really hard in practice. The main goal of this paper is proposing a robust approach for solving the above mentioned problems. Accordingly, this paper develops a novel framework for clustering problems, which is called Weighted Spectral Cluster Ensemble (WSCE), by exploiting some concepts from community detection arena and graph based clustering. Under this framework, a new version of spectral clustering, which is called Two Kernels Spectral Clustering, is used for generating graphs based individual clustering results. Further, by using modularity, which is a famous metric in the community detection, on the transformed graph representation of individual clustering results, our approach provides an effective diversity estimation for individual clustering results. Moreover, this paper introduces a new approach for combining the evaluated individual clustering results without the procedure of thresholding. Experimental study on varied data sets demonstrates that the prosed approach achieves superior performance to state-of-the-art methods.

Conventional learning-based methods for segmenting prostate in CT images ignore the relations among the low-level features by assuming all these features are independent. Also, their feature selection steps usually neglect the image appearance changes in different local regions of CT images. To this end, we present a novel semi-automatic learning-based prostate segmentation method in this article. For segmenting the prostate in a certain treatment image, the radiation oncologist will be first asked to take a few seconds to manually specify the first and last slices of the prostate. Then, prostate is segmented with the following two steps: (i) Estimation of 3D prostate-likelihood map to predict the likelihood of each voxel being prostate by employing the coupled feature representation, and the proposed Spatial-COnstrained Transductive LassO (SCOTO); (ii) Multi-atlases based label fusion to generate the final segmentation result by using the prostate shape information obtained from both planning and previous treatment images. The major contribution of the proposed method mainly includes: (i) incorporating radiation oncologist's manual specification to aid segmentation, (ii) adopting coupled features to relax previous assumption of feature independency for voxel representation, and (iii) developing SCOTO for joint feature selection across different local regions. The experimental result shows that the proposed method outperforms the state-of-the-art methods in a real-world prostate CT dataset, consisting of 24 patients with totally 330 images, all of which were manually delineated by the radiation oncologist for performance evaluation. Moreover, our method is also clinically feasible, since the segmentation performance can be improved by just requiring the radiation oncologist to spend only a few seconds for manual specification of ending slices in the current treatment CT image.

Presents the introductory editorial for this issue of the publication.

As thousands of features are available in many pattern recognition and machine learning applications, feature selection remains an important task to find the most compact representation of the original data. In the literature, although a number of feature selection methods have been developed, most of them focus on optimizing specific objective functions. In this paper, we first propose a general graph-preserving feature selection framework where graphs to be preserved vary in specific definitions, and show that a number of existing filter-type feature selection algorithms can be unified within this framework. Then, based on the proposed framework, a new filter-type feature selection method called sparsity score (SS) is proposed. This method aims to preserve the structure of a pre-defined l1graph that is proven robust to data noise. Here, the modified sparse representation based on an l1-norm minimization problem is used to determine the graph adjacency structure and corresponding affinity weight matrix simultaneously. Furthermore, a variant of SS called supervised SS (SuSS) is also proposed, where the l1graph to be preserved is constructed by using only data points from the same class. Experimental results of clustering and classification tasks on a series of benchmark data sets show that the proposed methods can achieve better performance than conventional filter-type feature selection methods.

Software defect prediction (SDP), which classifies software modules into defect-prone and not-defect-prone categories, provides an effective way to maintain high quality software systems. Most existing SDP models attempt to attain lower classification error rates other than lower misclassification costs. However, in many real-world applications, misclassifying defect-prone modules as not-defect-prone ones usually leads to higher costs than misclassifying not-defect-prone modules as defect-prone ones. In this paper, we first propose a new two-stage cost-sensitive learning (TSCS) method for SDP, by utilizing cost information not only in the classification stage but also in the feature selection stage. Then, specifically for the feature selection stage, we develop three novel cost-sensitive feature selection algorithms, namely, Cost-Sensitive Variance Score (CSVS), Cost-Sensitive Laplacian Score (CSLS), and Cost-Sensitive Constraint Score (CSCS), by incorporating cost information into traditional feature selection algorithms. The proposed methods are evaluated on seven real data sets from NASA projects. Experimental results suggest that our TSCS method achieves better performance in software defect prediction compared to existing single-stage cost-sensitive classifiers. Also, our experiments show that the proposed cost-sensitive feature selection methods outperform traditional cost-blind feature selection methods, validating the efficacy of using cost information in the feature selection stage.

Accurate prostate segmentation in CT images is a significant yet challenging task for image guided radiotherapy. In this paper, a novel semi-automated prostate segmentation method is presented. Specifically, to segment the prostate in the current treatment image, the physician first takes a few seconds to manually specify the first and last slices of the prostate in the image space. Then, the prostate is segmented automatically by the proposed two steps: (i) The first step of prostate-likelihood estimation to predict the prostate likelihood for each voxel in the current treatment image, aiming to generate the 3-D prostate-likelihood map by the proposed Spatial-COnstrained Transductive LassO (SCOTO), (ii) The second step of multi-atlases based label fusion to generate the final segmentation result by using the prostate shape information obtained from the planning and previous treatment images. The experimental result shows that the proposed method outperforms several state-of-the-art methods on prostate segmentation in a real prostate CT dataset, consisting of 24 patients with 330 images. Moreover, it is also clinically feasible since our method just requires the physician to spend a few seconds on manual specification of the first and last slices of the prostate.

Dimensionality reduction is an important task in the analysis of hyperspectral image data. While traditional dimensionality reduction methods use class labels as prior information, this letter presents a general semisupervised dimensionality reduction framework for hyperspectral image classification based on new prior information, i.e., pairwise constraints which specify whether a pair of examples belongs to the same class or not. The proposed semisupervised dimensionality reduction framework contains two terms: 1) a discrimination term that assesses the separability between classes; and 2) a regularization term that characterizes some property of the original data set. Furthermore, a novel semisupervised dimensionality reduction method is derived from the framework based on sparse representation. Experimental results on two hyperspectral image data sets show that the proposed algorithms are remarkably effective in comparison to traditional dimensionality reduction methods.

Traditional pattern recognition involves two tasks: clustering learning and classification learning. Clustering result can enhance the generalization ability of classification learning, while the class information can improve the accuracy of clustering learning. Hence, both learning methods can complement each other. To fuse the advantages of both learning methods together, many existing algorithms have been developed in a sequential fusing way by first optimizing the clustering criterion and then the classification criterion associated with the obtained clustering results. However, such kind of algorithms naturally fails to achieve the simultaneous optimality for two criteria, and thus has to sacrifice either the clustering performance or the classification performance. To overcome that problem, in this paper, we present a multiobjective simultaneous learning framework (MSCC) for both clustering and classification learning. MSCC utilizes multiple objective functions to formulate the clustering and classification problems, respectively, and more importantly, it employs the Bayesian theory to make these functions all only dependent on a set of the same parameters, i.e., clustering centers which play a role of the bridge connecting the clustering and classification learning. By simultaneously optimizing the clustering centers embedded in these functions, not only the effective clustering performance but also the promising classification performance can be simultaneously attained. Furthermore, from the multiple Pareto-optimality solutions obtained in MSCC, we can get an interesting observation that there is complementarity to great extent between clustering and classification learning processes. Empirical results on both synthetic and real data sets demonstrate the effectiveness and potential of MSCC.

The goal of semisupervised kernel matrix learning (SS-KML) is to learn a kernel matrix on all the given samples on which just a little supervised information, such as class label or pairwise constraint, is provided. Despite extensive research, the performance of SS-KML still leaves some space for improvement in terms of effectiveness and efficiency. For example, a recent pairwise constraints propagation (PCP) algorithm has formulated SS-KML into a semidefinite programming (SDP) problem, but its computation is very expensive, which undoubtedly restricts PCPs scalability in practice. In this paper, a novel algorithm, called kernel propagation (KP), is proposed to improve the comprehensive performance in SS-KML. The main idea of KP is first to learn a small-sized sub-kernel matrix (named seed-kernel matrix) and then propagate it into a larger-sized full-kernel matrix. Specifically, the implementation of KP consists of three stages: 1) separate the supervised sample (sub)set X(l) from the full sample set X; 2) learn a seed-kernel matrix on X(l) through solving a small-scale SDP problem; and 3) propagate the learnt seed-kernel matrix into a full-kernel matrix on X . Furthermore, following the idea in KP, we naturally develop two conveniently realizable out-of-sample extensions for KML: one is batch-style extension, and the other is online-style extension. The experiments demonstrate that KP is encouraging in both effectiveness and efficiency compared with three state-of-the-art algorithms and its related out-of-sample extensions are promising too.

The matrix, as an extended pattern representation to the vector, has proven to be effective in feature extraction. However, the subsequent classifier following the matrix-pattern- oriented feature extraction is generally still based on the vector pattern representation (namely, MatFE + VecCD), where it has been demonstrated that the effectiveness in classification just attributes to the matrix representation in feature extraction. This paper looks at the possibility of applying the matrix pattern representation to both feature extraction and classifier design. To this end, we propose a so-called fully matrixized approach, i.e., the matrix-pattern-oriented feature extraction followed by the matrix-pattern-oriented classifier design (MatFE + MatCD). To more comprehensively validate MatFE + MatCD, we further consider all the possible combinations of feature extraction (FE) and classifier design (CD) on the basis of patterns represented by matrix and vector respectively, i.e., MatFE + MatCD, MatFE + VecCD, just the matrix-pattern-oriented classifier design (MatCD), the vector-pattern-oriented feature extraction followed by the matrix-pattern-oriented classifier design (VecFE + MatCD), the vector-pattern-oriented feature extraction followed by the vector-pattern-oriented classifier design (VecFE + VecCD) and just the vector-pattern-oriented classifier design (VecCD). The experiments on the combinations have shown the following: 1) the designed fully matrixized approach (MatFE + MatCD) has an effective and efficient performance on those patterns with the prior structural knowledge such as images; and 2) the matrix gives us an alternative feasible pattern representation in feature extraction and classifier designs, and meanwhile provides a necessary validation for "ugly duckling" and "no free lunch" theorems.

Dimensionality reduction is among the keys in mining high-dimensional data. This paper studies semi-supervised dimensionality reduction. In this setting, besides abundant unlabeled examples, domain knowledge in the form of pairwise constraints are available, which specifies whether a pair of instances belong to the same class (must-link constraints) or different classes (cannot-link constraints). We propose the SSDR algorithm, which can preserve the intrinsic structure of the unlabeled data as well as both the must-link and cannot-link constraints defined on the labeled examples in the projected low-dimensional space. The SSDR algorithm is efficient and has a closed form solution. Experiments on a broad range of data sets show that SSDR is superior to many established dimensionality reduction methods.

Pseudoinverse Linear Discriminant Analysis (PLDA) is a classical and pioneer method that deals with the Small Sample Size (SSS) problem in LDA when applied to such applications as face recognition. However, it is expensive in computation and storage due to direct manipulation on extremely large d × d matrices, where d is the dimension of the sample image. As a result, although frequently cited in literature, PLDA is hardly compared in terms of classification performance with the newly proposed methods. In this paper, we propose a new feature extraction method named RSw + LDA, which is (1) much more efficient than PLDA in both computation and storage; and (2) theoretically equivalent to PLDA, meaning that it produces the same projection matrix as PLDA. Further, to make PLDA deal better with data of nonlinear distribution, we propose a Kernel PLDA (KPLDA) method with the well-known kernel trick. Finally, our experimental results on AR face dataset, a challenging dataset with variations in expression, lighting and occlusion, show that PLDA (or RSw + LDA) can achieve significantly higher classification accuracy than the recently proposed Linear Discriminant Analysis via QR decomposition and Discriminant Common Vectors, and KPLDA can yield better classification performance compared to PLDA and Kernel PCA.

The goal of this comment is to first point out two loopholes in the paper by Li (2006): (1) so-designed efficient maximal margin criterion (MMC) algorithm for small sample size (SSS) problem is problematic and (2) the discussion on the equivalence with the null-space-based methods in SSS problem does not hold. Then, we will present a really efficient MMC algorithm for SSS problem.

Choosing an appropriate kernel is one of the key problems in kernel-based methods. Most existing kernel selection methods require that the class labels of the training examples are known. In this paper, we propose an adaptive kernel selection method for kernel principal component analysis, which can effectively learn the kernels when the class labels of the training examples are not available. By iteratively optimizing a novel criterion, the proposed method can achieve nonlinear feature extraction and unsupervised kernel learning simultaneously. Moreover, a non-iterative approximate algorithm is developed. The effectiveness of the proposed algorithms are validated on UCI datasets and the COIL-20 object recognition database.

Fuzzy c-means clustering (FCM) with spatial constraints (FCM_S) is an effective algorithm suitable for image segmentation. Its effectiveness contributes not only to the introduction of fuzziness for belongingness of each pixel but also to exploitation of spatial contextual information. Although the contextual information can raise its insensitivity to noise to some extent, FCM_S still lacks enough robustness to noise and outliers and is not suitable for revealing non-Euclidean structure of the input data due to the use of Euclidean distance (L2 norm). In this paper, to overcome the above problems, we first propose two variants, FCM_S1 and FCM_S2, of FCM_S to aim at simplifying its computation and then extend them, including FCM_S, to corresponding robust kernelized versions KFCM_S, KFCM_S1 and KFCM_S2 by the kernel methods. Our main motives of using the kernel methods consist in: inducing a class of robust non-Euclidean distance measures for the original data space to derive new objective functions and thus clustering the non-Euclidean structures in data; enhancing robustness of the original clustering algorithms to noise and outliers, and still retaining computational simplicity. The experiments on the artificial and real-world datasets show that our proposed algorithms, especially with spatial constraints, are more effective.