Transfer learning is one of the popular methods to solve the problem of insufficient data in subject-specific electroencephalogram (EEG) recognition tasks. However, most existing approaches ignore the difference between subjects and transfer the same feature representations from source domain to different target domains, resulting in poor transfer performance. To address this issue, we propose a novel subject-specific EEG recognition method named deep multiview module adaption transfer (DMV-MAT) network. First, we design a universal deep multiview (DMV) network to generate different types of discriminative features from multiple perspectives, which improves the generalization performance by extensive feature sets. Second, module adaption transfer (MAT) is designed to evaluate each module by the feature distributions of source and target samples, which can generate an optimal weight sharing strategy for each target subject and promote the model to learn domain-invariant and domain-specific features simultaneously. We conduct extensive experiments in two EEG recognition tasks, i.e., motor imagery (MI) and seizure prediction, on four datasets. Experimental results demonstrate that the proposed method achieves promising performance compared with the state-of-the-art methods, indicating a feasible solution for subject-specific EEG recognition tasks. Implementation codes are available at https://github.com/YangLibuaa/DMV-MAT.

Cortical surface registration plays a crucial role in aligning cortical functional and anatomical features across individuals. However, conventional registration algorithms are computationally inefficient. Recently, learning-based registration algorithms have emerged as a promising solution, significantly improving processing efficiency. Nonetheless, there remains a gap in the development of a learning-based method that exceeds the state-of-the-art conventional methods simultaneously in computational efficiency, registration accuracy, and distortion control, despite the theoretically greater representational capabilities of deep learning approaches. To address the challenge, we present SUGAR, a unified unsupervised deep-learning framework for both rigid and non-rigid registration. SUGAR incorporates a U-Net-based spherical graph attention network and leverages the Euler angle representation for deformation. In addition to the similarity loss, we introduce fold and multiple distortion losses to preserve topology and minimize various types of distortions. Furthermore, we propose a data augmentation strategy specifically tailored for spherical surface registration to enhance the registration performance. Through extensive evaluation involving over 10,000 scans from 7 diverse datasets, we showed that our framework exhibits comparable or superior registration performance in accuracy, distortion, and test-retest reliability compared to conventional and learning-based methods. Additionally, SUGAR achieves remarkable sub-second processing times, offering a notable speed-up of approximately 12,000 times in registering 9,000 subjects from the UK Biobank dataset in just 32 min. This combination of high registration performance and accelerated processing time may greatly benefit large-scale neuroimaging studies.

Emotion recognition in conversation (ERC) based on multiple modalities has attracted enormous attention. However, most research simply concatenated multimodal representations, generally neglecting the impact of cross-modal correspondences and uncertain factors, and leading to the cross-modal misalignment problems. Furthermore, recent methods only considered simple contextual features, commonly ignoring semantic clues and resulting in an insufficient capture of the semantic consistency. To address these limitations, we propose a novel multi-level alignment and cross-modal unified semantic graph refinement network (MA-CMU-SGRNet) for ERC task. Specifically, a multi-level alignment (MA) is first designed to bridge the gap between acoustic and lexical modalities, which can effectively contrast both the instance-level and prototype-level relationships, separating the multimodal features in the latent space. Second, a cross-modal uncertainty-aware unification (CMU) is adopted to generate a unified representation in joint space considering the ambiguity of emotion. Finally, a dual-encoding semantic graph refinement network (SGRNet) is investigated, which includes a syntactic encoder to aggregate information from near neighbors and a semantic encoder to focus on useful semantically close neighbors. Extensive experiments on three multimodal public datasets show the effectiveness of our proposed method compared with the state-of-the-art methods, indicating its potential application in conversational emotion recognition. Implementation codes can be available athttps://github.com/zxiaohen/MA-CMU-SGRNet.

Detecting and diagnosing major depressive disorder (MDD) is greatly crucial for appropriate treatment and support. In recent years, there have been efforts to develop automated methods for depression detection using machine learning techniques, which mainly analyze various data sources such as text, speech, and social media posts. However, the effectiveness and reliability of these methods may vary and more importantly, they fail to provide timely intervention and treatment to MDD patients. To address these challenges, we propose a novel electroencephalogram (EEG)-based MDD detection framework, which is named as multiview sparse dynamic graph convolution-based region-attention feature fusion network (MV-SDGC-RAFFNet). Specifically, we first design a multiview (MV) feature extractor to concurrently characterize EEG signals from temporal, spectral, and time-frequency views, providing rich semantic information on the emotional status of patients. Secondly, we introduce a sparse dynamic graph convolution network (SDGCN) to map the multidomain features into high-level representations, which avoids the limitation of over-smoothing and redundant edges existing in the conventional graph neural networks (GNNs). Finally, to efficiently fuse multidomain features, we propose a region-attention feature fusion network (RAFFNet), which applies different attention weights for brain regions and is greatly beneficial to boost the accuracy (ACC) of MDD detection. We validate the efficacy of the proposed MV-SDGC-RAFFNet framework on two public MDD datasets, and it achieves more promising detection performance against the state-of-the-art methods, indicating that our method has a prospect on clinical MDD detection.

Recognizing visual objects from single-trial electroencephalograph (EEG) signals is a promising brain-computer interface technology. However, due to the redundant features from noisy multichannel EEG signals, it is still a challenging task to achieve high precision recognition. Recent deep learning approaches commonly extract spatio-temporal features of EEG signals, which neglect important spectral-temporal features and may degrade the EEG recognition performance. To address the deficiency, we propose a novel channel attention weighting and multilevel adaptive spectral aggregation based dual-branch spatio-temporal-spectral transformer feature fusion network (CAW-MASA-STST) for EEG-based visual recognition. Specially, we first develop a channel attention weighting (CAW) to automatically learn the channel weights of EEG signals. Then, a graph convolution-based MASA is employed to aggregate spectral-temporal features of different sub-bands. Finally, an STST is designed to fuse spatio-temporal and spectral-temporal features, which enhances the comprehensive learning ability by modeling the temporal dependencies of the fused features. Competitive experimental results on two public datasets demonstrate that the proposed method is able to achieve superior recognition performance compared with the state-of-the-art methods, indicating a feasible solution for visual recognition-based BCI technology. The code of our proposed method will be available athttps://github.com/ljbuaa/VisualDecoding.

Conventional functional connectivity network (FCN) based on resting-state fMRI (rs-fMRI) can only reflect the relationship between pairwise brain regions. Thus, the hyper-connectivity network (HCN) has been widely used to reveal high-order interactions among multiple brain regions. However, existing HCN models are essentially spatial HCN, which reflect the spatial relevance of multiple brain regions, but ignore the temporal correlation among multiple time points. Furthermore, the majority of HCN construction and learning frameworks are limited to using a single template, while the multi-template carries richer information. To address these issues, we first employ multiple templates to parcellate the rs-fMRI into different brain regions. Then, based on the multi-template data, we propose a spatio-temporal weighted HCN (STW-HCN) to capture more comprehensive high-order temporal and spatial properties of brain activity. Next, a novel deep fusion model of multi-template called spatio-temporal weighted multi-hypergraph convolutional network (STW-MHGCN) is proposed to fuse the STW-HCN of multiple templates, which extracts the deep interrelation information between different templates. Finally, we evaluate our method on the ADNI-2 and ABIDE-I datasets for mild cognitive impairment (MCI) and autism spectrum disorder (ASD) analysis. Experimental results demonstrate that the proposed method is superior to the state-of-the-art approaches in MCI and ASD classification, and the abnormal spatio-temporal hyper-edges discovered by our method have significant significance for the brain abnormalities analysis of MCI and ASD.

Functional connectivity (FC) networks based on resting-state functional magnetic imaging (rs-fMRI) are reliable and sensitive for brain disorder diagnosis. However, most existing methods are limited by using a single template, which may be insufficient to reveal complex brain connectivities. Furthermore, these methods usually neglect the complementary information between static and dynamic brain networks, and the functional divergence among different brain regions, leading to suboptimal diagnosis performance. To address these limitations, we propose a novel multi-graph cross-attention based region-aware feature fusion network (MGCA-RAFFNet) by using multi-template for brain disorder diagnosis. Specifically, we first employ multi-template to parcellate the brain space into different regions of interest (ROIs). Then, a multi-graph cross-attention network (MGCAN), including static and dynamic graph convolutions, is developed to explore the deep features contained in multi-template data, which can effectively analyze complex interaction patterns of brain networks for each template, and further adopt a dual-view cross-attention (DVCA) to acquire complementary information. Finally, to efficiently fuse multiple static-dynamic features, we design a region-aware feature fusion network (RAFFNet), which is beneficial to improve the feature discrimination by considering the underlying relations among static-dynamic features in different brain regions. Our proposed method is evaluated on both public ADNI-2 and ABIDE-I datasets for diagnosing mild cognitive impairment (MCI) and autism spectrum disorder (ASD). Extensive experiments demonstrate that the proposed method outperforms the state-of-the-art methods. Our source code is available at https://github.com/mylbuaa/MGCA-RAFFNet.

Facial expression recognition (FER) in the wild is a challenging task for affective computing in human–machine interaction fields. However, most of the existing methods fail to learn the most prominent regions of facial images by simple cross-entropy loss due to the imbalance problem commonly existing in FER datasets, which limits the robustness and interpretability of the model. In addition, these methods only capture local features of original images with multisize shallow convolution and ignore facial texture characteristics, leading to a suboptimal recognition performance. To address these issues, in this article, we propose a novel FER network, named the attention-rectified and texture-enhanced cross-attention transformer feature fusion network (AR-TE-CATFFNet). Specifically, an attention-rectified convolution block is first designed to assist multiple convolution heads to focus on the critical areas of human faces and improve the model generalization. Second, we investigate a texture enhancement block to capture texture features through local binary pattern and gray-level co-occurrence matrix, which solves the limitation of insufficient texture information. Finally, a cross-attention transformer feature fusion block is employed to deeply integrate red, green, blue (RGB) features and texture features globally, which is beneficial to boost the accuracy of recognition. Competitive experimental results on three public datasets validate the efficacy of the proposed method, indicating that our proposed method achieves superior classification performance of 89.50% on real-world affective faces database (RAF-DB) dataset, 65.66% on AffectNet dataset, and 74.84% on FER2013 dataset against the existing methods.

Electroencephalograph (EEG) emotion recognition plays an important role in the brain-computer interface (BCI) field. However, most of recent methods adopted shallow graph neural networks using a single temporal feature, leading to the limited emotion classification performance. Furthermore, the existing methods generally ignore the individual divergence between different subjects, resulting in poor transfer performance. To address these deficiencies, we propose a dual-branch dynamic graph convolution based adaptive transformer feature fusion network with adapter-finetuned transfer learning (DBGC-ATFFNet-AFTL) for EEG emotion recognition. Specifically, a dual-branch graph convolution network (DBGCN) is firstly designed to effectively capture the temporal and spectral characterizations of EEG simultaneously. Second, the adaptive Transformer feature fusion network (ATFFNet) is conducted by integrating the obtained feature maps with the channel-weight unit, leading to significant difference between different channels. Finally, the adapter-finetuned transfer learning method (AFTL) is applied in cross-subject emotion recognition, which proves to be parameter-efficient with few samples of the target subject. The competitive experimental results on three datasets have shown that our proposed method achieves the promising emotion classification performance compared with the state-of-the-art methods. The code of our proposed method will be available at:https://github.com/smy17/DANet.

Retinal vessel segmentation with deep learning technology is a crucial auxiliary method for clinicians to diagnose fundus diseases. However, the deep learning approaches inevitably lose the edge information, which contains spatial features of vessels while performing down-sampling, leading to the limited segmentation performance of fine blood vessels. Furthermore, the existing methods ignore the dynamic topological correlations among feature maps in the deep learning framework, resulting in the inefficient capture of the channel characterization. To address these limitations, we propose a novel dual encoder-based dynamic-channel graph convolutional network with edge enhancement (DE-DCGCN-EE) for retinal vessel segmentation. Specifically, we first design an edge detection-based dual encoder to preserve the edge of vessels in down-sampling. Secondly, we investigate a dynamic-channel graph convolutional network to map the image channels to the topological space and synthesize the features of each channel on the topological map, which solves the limitation of insufficient channel information utilization. Finally, we study an edge enhancement block, aiming to fuse the edge and spatial features in the dual encoder, which is beneficial to improve the accuracy of fine blood vessel segmentation. Competitive experimental results on five retinal image datasets validate the efficacy of the proposed DE-DCGCN-EE, which achieves more remarkable segmentation results against the other state-of-the-art methods, indicating its potential clinical application.

The electroencephalogram (EEG) signal analysis is a valuable tool in the evaluation of neurological disorders, which is commonly used for the diagnosis of epileptic seizures. This paper presents a novel automatic EEG signal classification method for epileptic seizure detection. The proposed method first employs a continuous wavelet transform (CWT) method for obtaining the time-frequency images (TFI) of EEG signals. The processed EEG signals are then decomposed into five sub-band frequency components of clinical interest since these sub-band frequency components indicate much better discriminative characteristics. Both Gaussian Mixture Model (GMM) features and Gray Level Co-occurrence Matrix (GLCM) descriptors are then extracted from these sub-band TFI. Additionally, in order to improve classification accuracy, a compact feature selection method by combining the ReliefF and the support vector machine-based recursive feature elimination (RFE-SVM) algorithm is adopted to select the most discriminative feature subset, which is an input to the SVM with the radial basis function (RBF) for classifying epileptic seizure EEG signals. The experimental results from a publicly available benchmark database demonstrate that the proposed approach provides better classification accuracy than the recently proposed methods in the literature, indicating the effectiveness of the proposed method in the detection of epileptic seizures.

A new parametric approach is proposed for nonlinear and nonstationary system identification based on a time-varying nonlinear autoregressive with exogenous input (TV-NARX) model. The TV coefficients of the TV-NARX model are expanded using multiwavelet basis functions, and the model is thus transformed into a time-invariant regression problem. An ultra-orthogonal forward regression (UOFR) algorithm aided by mutual information (MI) is designed to identify a parsimonious model structure and estimate the associated model parameters. The UOFR-MI algorithm, which uses not only the observed data themselves but also weak derivatives of the signals, is more powerful in model structure detection. The proposed approach combining the advantages of both the basis function expansion method and the UOFR-MI algorithm is proved to be capable of tracking the change of TV parameters effectively in both numerical simulations and the real EEG data.