Emotion Recognition in Conversation (ERC) is valuable for diagnosing health conditions such as autism and depression (Maryenko, 2024), and for understanding the emotions of individuals who struggle to express their feelings. Current ERC methods primarily rely on semantic, audio and video data but face significant challenges in integrating physiological signals such as Electroencephalography (EEG), which has low signal-to-noise ratios, inter-subject variability, and temporal alignment issues. This research proposes Hypergraph Multi-Modal Learning (Hyper-MML), a novel framework for identifying emotions in conversation. Hyper-MML effectively integrates EEG with audio and video information to capture complex emotional dynamics. Firstly, we introduce an Adaptive Brain Encoder with Mutual-cross Attention (ABEMA) module for processing EEG signals. This module captures emotion-relevant features across different frequency bands and adapts to subject-specific variations through hierarchical mutual-cross attention mechanisms. Secondly, we propose an Adaptive Hypergraph Fusion Module (AHFM) to actively model the higher-order relationships among multi-modal signals in ERC. Experimental results on the EAV and AFFEC datasets demonstrate that our Hyper-MML model significantly outperforms current state-of-the-art methods. The proposed Hyper-MML can serve as an effective communication tool for healthcare professionals, enabling better engagement with patients who have difficulty expressing their emotions. The official implementation codes are available at https://github.com/NZWANG/Hyper-MML.

Most functional magnetic resonance imaging studies rely on estimates of hierarchically organized functional brain networks whose segregation and integration reflect the cognitive and behavioral changes in humans. However, most existing methods for estimating the community structure of networks from both individual and group-level analysis methods do not account for the variability between subjects. In this paper, we develop a new multilayer community detection method based on Bayesian latent block model (LBM). The method can robustly detect the community structure of weighted functional networks with an unknown number of communities at both individual and group levels and retain the variability of the individual networks. For validation, we propose a new community structure-based multivariate Gaussian generative model to simulate synthetic signal. Our simulation study shows that the community memberships estimated by hierarchical Bayesian inference are consistent with the predefined node labels in the generative model. The method is also tested via split-half reproducibility using working memory task fMRI data of 100 unrelated healthy subjects from the Human Connectome Project. Analyses using both synthetic and real data show that our proposed method is more accurate and reliable compared with the commonly used (multilayer) modularity models. The code of this work is available at: https://github.com/LingbinBian/CommuDetectLBM.

The advancement of single-cell sequencing technology has provided unprecedented resolution for investigating cellular heterogeneity. Methods based on non-negative matrix factorization (NMF) and autoencoders are widely applied in single-cell sequencing analysis. However, current analytical models for single-cell sequencing data still face challenges such as high noise and limited applicability to specific scenarios, leading to suboptimal clustering performance. To address this issue, this study proposes an Autoencoder-like Dual-Graph Nonnegative Matrix Factorization (ADGNMF) model for single-cell multiomics analysis. The proposed method first modifies the joint NMF into an autoencoder-like architecture, followed by construction of multi-omics graph regularization and co-cluster graph regularization to enhance clustering performance and representational capability of the model. Experimental results on 8 multi-source transcriptomic datasets, 2 transcriptomic-epigenomic datasets, and 2 transcriptomic-proteomic datasets validate superior clustering performance and biological interpretability of the model. The source code of ADGNMF is available at https://github.com/jj-LanJADGNMF.

Decoding neural visual representations from electroencephalogram (EEG)-based brain activity is crucial for advancing brain-machine interfaces (BMI) and has transformative potential for neural sensory rehabilitation. While multimodal contrastive representation learning (MCRL) has shown promise in neural decoding, existing methods often overlook semantic consistency and completeness within modalities and lack effective semantic alignment across modalities. This limits their ability to capture the complex representations of visual neural responses. We propose Neural-MCRL, a novel framework that achieves multimodal alignment through semantic bridging and cross-attention mechanisms, while ensuring completeness within modalities and consistency across modalities. Our framework also features the Neural Encoder with Spectral-Temporal Adaptation (NESTA), a EEG encoder that adaptively captures spectral patterns and learns subject-specific transformations. Experimental results demonstrate significant improvements in visual decoding accuracy and model generalization compared to state-of-the-art methods, advancing the field of EEG-based neural visual representation decoding in BMI. Code will be available at: https://github.com/NZWANG/Neural-MCRL.

Graph deep learning (GDL) has demonstrated impressive performance in predicting population-based brain disorders (BDs) through the integration of both imaging and non-imaging data. However, the effectiveness of GDL-based methods heavily depends on the quality of modeling multi-modal population graphs and tends to degrade as the graph scale increases. Moreover, these methods often limit interactions between imaging and non-imaging data to node-edge interactions within the graph, overlooking complex inter-modal correlations and resulting in suboptimal outcomes. To address these challenges, we propose MM-GTUNets, an end-to-end Graph Transformer-based multi-modal graph deep learning (MMGDL) framework designed for large-scale brain disorders prediction. To effectively utilize rich multi-modal disease-related information, we introduce Modality Reward Representation Learning (MRRL), which dynamically constructs population graphs using an Affinity Metric Reward System (AMRS). We also employ a variational autoencoder to reconstruct latent representations of non-imaging features aligned with imaging features. Based on this, we introduce Adaptive Cross-Modal Graph Learning (ACMGL), which captures critical modality-specific and modality-shared features through a unified GTUNet encoder, taking advantages of Graph UNet and Graph Transformer, along with a feature fusion module. We validated our method on two public multi-modal datasets ABIDE and ADHD-200, demonstrating its superior performance in diagnosing BDs. Our code is available at https://github.com/NZWANG/MM-GTUNetshttps://github.com/NZWANG/MM-GTUNets.

Accurate segmentation of prostate tumors from multi-modal magnetic resonance (MR) images is crucial for diagnosis and treatment of prostate cancer. However, the robustness of existing segmentation methods is limited, mainly because these methods 1) fail to adaptively assess subject-specific information of each MR modality for accurate tumor delineation, and 2) lack effective utilization of inter-slice information across thick slices in MR images to segment tumor as a whole 3D volume. In this work, we propose a two-stage neighbor-aware multi-modal adaptive learning network (NaMa) for accurate prostate tumor segmentation from multi-modal anisotropic MR images. In particular, in the first stage, we apply subject-specific multi-modal fusion in each slice by developing a novel modality-informativeness adaptive learning (MIAL) module for selecting and adaptively fusing informative representation of each modality based on inter-modality correlations. In the second stage, we exploit inter-slice feature correlations to derive volumetric tumor segmentation. Specifically, we first use a Unet variant with sequence layers to coarsely capture slice relationship at a global scale, and further generate an activation map for each slice. Then, we introduce an activation mapping guidance (AMG) module to refine slice-wise representation (via information from adjacent slices) for consistent tumor segmentation across neighboring slices. Besides, during the network training, we further apply a random mask strategy to each MR modality to improve feature representation efficiency. Experiments on both in-house and public (PICAI) multi-modal prostate tumor datasets show that our proposed NaMa performs better than state-of-the-art methods.

Independent component analysis aiming at detecting the functional connectivity among discrete cortical brain regions has been extensively used to explore the functional magnetic resonance imaging data. Although the independent components (ICs) were with relatively high quality, the noise embedding in ICs has a great impact on the true active/inactive region inference and the reproducibility, in postprocessing stage, e.g., the extraction of statistical parametrical maps (SPMs). In this paper, a novel brain network enhancement model (BNEM) is proposed, which mainly consists of two key techniques: 1) 3-D wavelet noise filter (3DWNF) for the meaningful ICs, which greatly suppresses noise and enforces the real activation inference of SPMs; and 2) a spatial reproducibility enhancement algorithm (SREA), aiming to improve the reproducibility of SPMs. The simulated experiment demonstrated that the postfiltering signals by 3DWNF were with higher correlation and less normalized mean square error to the ground truths than the prefiltering ones; SREA could further enhance the quality of most postfiltering ones, preserving the consistency with 3DWNF. The real data experiments also revealed that 1) 3DWNF could lead to more accurate preservation of the true positive voxels by correctly identifying the high proportionally misclassified voxels of the nonenhanced SPMs; 2) SREA could further improve the classification accuracy of the active/inactive voxels of SPMs corresponding to the 3DWNF denoised ICs; and 3) both 3DWNF and SREA contribute to the reproducibility enhancement of the reproduced SPMs by BNEM. Thus, BNEM is expected to have wide applicability in the neuroscience and clinical domain.