A spontaneous electroencephalogram (EEG)-based brain-computer interface (BCI) is an ideal form of brain-computer interaction. The classical decoding methods can achieve classification by using meaningful manual features, but their performance is poor. The neural network (NN) methods have significantly improved the performance, but their interpretability and computational efficiency are much lower than those of the classical methods. This is because NN abandons the strong a priori knowledge of neuroscience and completely relies on training to extract EEG features. How to integrate the characteristics of neural signals into the design of the basic operator of the NNs while retaining its learning ability is the focus of this work. In this work, we proposed a function predefined convolutional NN (FPCNN) to search for the best frequency points and channel weights to decode spontaneous EEG signals. Among the FPCNN, a novel function predefined convolutional (FPC) layer adopts a learnable way to search for the key spatial-frequency parameters of spontaneous EEG, making its parameters have clear physical meanings. Furthermore, a trainable quadrature detector (TQD) based on FPC was constructed, and the quadrature characteristic was utilized to ensure the capture of complex phase change signals. The core contribution of our method lies in the proposal of a novel NN operator for decoding spontaneous EEG, and a quadrature scheme for handling the phase changes of signals. The experimental results show that the proposed FPCNN significantly improves the performance by 2.09% ( ${}^{\ast } $ ), 3.08% ( ${}^{\ast } $ ), and 3.41% ( ${}^{\ast \ast }$ ), respectively, compared with the state-of-the-art (SOTA) methods on three spontaneous EEG datasets. Moreover, the training and testing time cost of FPCNN in a non-GPU environment only takes 67.96 and 19.36 s per epoch. Its savings in computing resources and time are very beneficial for EEG processing in diverse environments. In addition, visualization experiments demonstrated the interpretability and stability of the proposed FPCNN. The experimental results show that our method is efficient, stable, and interpretable. This work has effectively improved the decoding efficiency of spontaneous EEG signals and demonstrated the power of combining traditional signal processing methods with NNs.

In recent years, numerous neuroscientific studies have demonstrated that specific brain regions are associated with human emotional responses, with these regions exhibiting variability across individuals and emotions. To effectively leverage these neural patterns, we propose an Adaptive Progressive Attention Graph Neural Network (APAGNN), which dynamically models the spatial relationships among brain regions during emotional processing. APAGNN employs three specialized expert modules that progressively analyze brain topology. The first expert captures global brain connectivity patterns, the second extracts localized regional features, and the third focuses on emotion-related channel interactions. This progressive refinement strategy enables hierarchical feature extraction from coarsegrained to fine-grained neural representations. Furthermore, a weight generator integrates the outputs of all three experts, adaptively balancing their contributions for final emotion recognition. Extensive experiments conducted on SEED, SEED-IV and MPED datasets demonstrate that our method significantly improves EEG emotion recognition performance, achieving superior results compared to baseline methods.

Graph neural networks (GNNs) are gaining increasing popularity for EEG-based depression detection. However, previous GNN-based methods inadequately consider the characteristics of depression, which limit their performance. First, neuroscience studies indicate that patients with depression exhibit both common and individualized brain abnormalities. Previous GNN-based approaches typically focus either on common graph connections to capture common brain abnormalities or on individualized connections to capture individualized patterns, which is insufficient for depression detection. Second, brain network exhibits a hierarchical structure, ranging from channel-level graphs to region-level graphs. This hierarchical structure varies across individuals and contains significant information relevant to detecting depression. However, previous GNN-based methods overlook this individualized hierarchical information. To address these issues, we propose a Hybrid GNN (HybGNN) that combines a Common Graph Neural Network (CGNN) branch using common connections and an Individualized Graph Neural Network (IGNN) branch employing individualized connections. The two branches capture common and individualized depression patterns, respectively, complementing each other. Furthermore, we enhance the HybGNN with a Cross-Branch Hierarchical Information Extractor (CB-HIE) to extract more task-relevant individualized hierarchical information. Extensive experiments on the MODMA and HUSM datasets demonstrate that the proposed HybGNN achieves state-of-the-art performance.

Electroencephalogram (EEG)-based emotion recognition has achieved remarkable success in both subject-dependent and subject-independent scenarios. However, overcoming the challenges associated with reduced performance in EEG emotion recognition across devices, time, space, and subjects (i.e., cross-dataset) remains a significant obstacle for affective brain-computer interfaces (aBCIs). The key issue lies in the distributional mismatch between source and target domain EEG signals. To tackle the significant inter-domain differences in cross-dataset EEG emotion recognition, this paper introduces an innovative framework termed the Emotional EEG Style Transfer Network (E$^{2}$STN), which aims to effectively capture the emotional content information from the source domain and the style features from the target domain, facilitating the reconstruction of stylized emotion EEG representations. These stylized EEG representations significantly enhance the discriminative prediction performance in cross-dataset EEG emotion recognition. Specifically, E$^{2}$STN consists of three key modules: a Transfer Module for domain style transfer, a Transfer Evaluation Module for evaluating transfer quality, and a Discriminative Module for making discriminative predictions. Extensive experiments demonstrate that E$^{2}$STN achieves the state-of-the-art performance in cross-dataset emotion EEG recognition. To the best of our knowledge, this is the first work to explicitly address cross-dataset emotion EEG recognition. The experimental results provide a valuable benchmark for future research in this area.

Previous electroencephalogram (EEG) emotion recognition relies on single-task learning, which may lead to overfitting and learned emotion features lacking generalization. In this paper, a graph-based multi-task self-supervised learning model (GMSS) for EEG emotion recognition is proposed. GMSS has the ability to learn more general representations by integrating multiple self-supervised tasks, including spatial and frequency jigsaw puzzle tasks, and contrastive learning tasks. By learning from multiple tasks simultaneously, GMSS can find a representation that captures all of the tasks thereby decreasing the chance of overfitting on the original task, i.e., emotion recognition task. In particular, the spatial jigsaw puzzle task aims to capture the intrinsic spatial relationships of different brain regions. Considering the importance of frequency information in EEG emotional signals, the goal of the frequency jigsaw puzzle task is to explore the crucial frequency bands for EEG emotion recognition. To further regularize the learned features and encourage the network to learn inherent representations, contrastive learning task is adopted in this work by mapping the transformed data into a common feature space. The performance of the proposed GMSS is compared with several popular unsupervised and supervised methods. Experiments on SEED, SEED-IV, and MPED datasets show that the proposed model has remarkable advantages in learning more discriminative and general features for EEG emotional signals.

The individual differences and the dynamic uncertain relationships among different electroencephalogram (EEG) regions are essential factors that limit EEG emotion recognition. To address these issues, in this article, we propose a variational instance-adaptive graph method (V-IAG) that simultaneously captures the individual dependencies among different EEG electrodes and estimates the underlying uncertain information. Specifically, we employ two branches, i.e., instance-adaptive branch and variational branch, to construct the graph. Inspired by the attention mechanism, the instance-adaptive branch generates the graph based on the input so as to characterize the individual dependencies among EEG channels. The variational branch generates the probabilistic graph, which quantifies the uncertainties. We combine these two types of graphs to extract more discriminative features. To present more precise graph representation, we propose a new operation named the multi-level and multi-graph convolution operation, which aggregates the features of EEG channels from different frequencies with different graphs. Furthermore, we design the graph coarsening and employ the sparse constraint to obtain more robust features. We conduct extensive experiments on three widely-used EEG emotion recognition databases, i.e., SJTU emotion EEG dataset (SEED), multi-modal physiological emotion recognition dataset (MPED) and DREAMER. The results demonstrate that the proposed model achieves the-state-of-the-art performance.

In this paper, we propose a novel Electroencephalograph (EEG) emotion recognition method inspired by neuroscience with respect to the brain response to different emotions. The proposed method, denoted by R2G-STNN, consists of spatial and temporal neural network models with regional to global hierarchical feature learning process to learn discriminative spatial-temporal EEG features. To learn the spatial features, a bidirectional long short term memory (BiLSTM) network is adopted to capture the intrinsic spatial relationships of EEG electrodes within brain region and between brain regions, respectively. Considering that different brain regions play different roles in the EEG emotion recognition, a region-attention layer into the R2G-STNN model is also introduced to learn a set of weights to strengthen or weaken the contributions of brain regions. Based on the spatial feature sequences, BiLSTM is adopted to learn both regional and global spatial-temporal features and the features are fitted into a classifier layer for learning emotion-discriminative features, in which a domain discriminator working corporately with the classifier is used to decrease the domain shift between training and testing data. Finally, to evaluate the proposed method, we conduct both subject-dependent and subject-independent EEG emotion recognition experiments on SEED database, and the experimental results show that the proposed method achieves state-of-the-art performance.

In this paper, we propose a novel domain invariant feature learning (DIFL) method to deal with speaker-independent speech emotion recognition (SER). The basic idea of DIFL is to learn the speaker-invariant emotion feature by eliminating domain shifts between the training and testing data caused by different speakers from the perspective of multi-source unsupervised domain adaptation (UDA). Specifically, we embed a hierarchical alignment layer with the strong-weak distribution alignment strategy into the feature extraction block to firstly reduce the discrepancy in feature distributions of speech samples across different speakers as much as possible. Furthermore, multiple discriminators in the discriminator block are utilized to confuse the speaker information of emotion features both inside the training data and between the training and testing data. Through them, a multi-domain invariant representation of emotional speech can be gradually and adaptively achieved by updating network parameters. We conduct extensive experiments on three public datasets, i. e., Emo-DB, eNTERFACE, and CASIA, to evaluate the SER performance of the proposed method, respectively. The experimental results show that the proposed method is superior to the state-of-the-art methods.

In this paper, we propose a novel neural network model, called bi-hemisphere domain adversarial neural network (BiDANN) model, for electroencephalograph (EEG) emotion recognition. The BiDANN model is inspired by the neuroscience findings that the left and right hemispheres of human's brain are asymmetric to the emotional response. It contains a global and two local domain discriminators that work adversarially with a classifier to learn discriminative emotional features for each hemisphere. At the same time, it tries to reduce the possible domain differences in each hemisphere between the source and target domains so as to improve the generality of the recognition model. In addition, we also propose an improved version of BiDANN, denoted by BiDANN-S, for subject-independent EEG emotion recognition problem by lowering the influences of the personal information of subjects to the EEG emotion recognition. Extensive experiments on the SEED database are conducted to evaluate the performance of both BiDANN and BiDANN-S. The experimental results have shown that the proposed BiDANN and BiDANN models achieve state-of-the-art performance in the EEG emotion recognition.

In this paper, we propose a novel deep learning framework, called spatial-temporal recurrent neural network (STRNN), to integrate the feature learning from both spatial and temporal information of signal sources into a unified spatial-temporal dependency model. In STRNN, to capture those spatially co-occurrent variations of human emotions, a multidirectional recurrent neural network (RNN) layer is employed to capture long-range contextual cues by traversing the spatial regions of each temporal slice along different directions. Then a bi-directional temporal RNN layer is further used to learn the discriminative features characterizing the temporal dependencies of the sequences, where sequences are produced from the spatial RNN layer. To further select those salient regions with more discriminative ability for emotion recognition, we impose sparse projection onto those hidden states of spatial and temporal domains to improve the model discriminant ability. Consequently, the proposed two-layer RNN model provides an effective way to make use of both spatial and temporal dependencies of the input signals for emotion recognition. Experimental results on the public emotion datasets of electroencephalogram and facial expression demonstrate the proposed STRNN method is more competitive over those state-of-the-art methods.

In this paper, we propose a novel neural network model, called bi-hemispheres domain adversarial neural network (BiDANN), for EEG emotion recognition. BiDANN is motivated by the neuroscience findings, i.e., the emotional brain's asymmetries between left and right hemispheres. The basic idea of BiDANN is to map the EEG feature data of both left and right hemispheres into discriminative feature spaces separately, in which the data representations can be classified easily. For further precisely predicting the class labels of testing data, we narrow the distribution shift between training and testing data by using a global and two local domain discriminators, which work adversarially to the classifier to encourage domain-invariant data representations to emerge. After that, the learned classifier from labeled training data can be applied to unlabeled testing data naturally. We conduct two experiments to verify the performance of our BiDANN model on SEED database. The experimental results show that the proposed model achieves the state-of-the-art performance.

Automatic Facial Action Unit (AU) detection has drawn more and more attention over the past years due to its significance to facial expression analysis. Frontal-view AU detection has been extensively evaluated, but cross-pose AU detection is a less-touched problem due to the scarcity of the related dataset. The challenge of Facial Expression Recognition and Analysis (FERA2017) just released a large-scale videobased AU detection dataset across different facial poses. To deal with this challenging task, we develop a simple and efficient deep learning based system to detect AU occurrence under nine different facial views. In this system, we first crop out facial images by using morphology operations including binary segmentation, connected components labeling and region boundaries extraction, then for each type of AU, we train a corresponding expert network by specifically fine-tuning the VGG-Face network on cross-view facial images, so as to extract more discriminative features for the subsequent binary classification. In the AU detection sub-challenge, our proposed method achieves the mean accuracy of 77.8% (vs. the baseline 56.1%), and promotes the F1 score to 57.4% (vs. the baseline 45.2%).