Electroencephalogram (EEG)-based emotion decoding can objectively quantify people's emotional state and has broad application prospects in human-computer interaction and early detection of emotional disorders. Recently emerging deep learning architectures have significantly improved the performance of EEG emotion decoding. However, existing methods still fall short of fully capturing the complex spatiotemporal dynamics of neural signals, which are crucial for representing emotion processing. This study proposes a Dynamic-Attention-based EEG State Transition (DAEST) modeling method to characterize EEG spatiotemporal dynamics. The model extracts spatiotemporal components of EEG that represent multiple parallel neural processes and estimates dynamic attention weights on these components to capture transitions in brain states. The model is optimized within a contrastive learning framework for cross-subject emotion recognition. The proposed method achieved state-of-the-art performance on three publicly available datasets: FACED, SEED, and SEED-V. It achieved$81.7\pm 4.3\%$accuracy in the binary classification of positive and negative emotions and$67.9\pm 7.3\%$in nine-class discrete emotion classification on the FACED dataset,$88.1\pm 3.6\%$in the three-class classification of positive, negative, and neutral emotions on the SEED dataset, and$73.6\pm 12.7\%$in five-class discrete emotion classification on the SEED-V dataset. The learned EEG spatiotemporal patterns and dynamic transition properties offer valuable insights into neural dynamics underlying emotion processing.

Virtual Reality (VR) technologies in fields such as telehealth, teleconferencing, and virtual education are significantly affected by end-to-end latency, which notably impacts users' interactive experience and performance. Previous research suggests that a perceptual threshold may exist-once latency is reduced below a certain level, users no longer perceive it, and their interactive performance remains largely unaffected. However, there is no consensus on the exact value of this absolute latency perception threshold. In this study, we employed an experimental design based on Fitts' law to investigate whether interaction strategies and task difficulty can alter the latency perception threshold (LPT), and how variations in this threshold influence users' interactive performance. The results show that the LPT is approximately 130-170 ms, and that when interaction strategies prioritize speed or when tasks become more challenging, users exhibit heightened sensitivity to latency. Due to the presence of the LPT, the effect of latency on interactive performance follows a nonlinear pattern, and building on this finding, we refined a Fitts' law model to incorporate the influence of latency. Notably, electroencephalogram (EEG) signals can still capture users' perception of latency when they are unaware of minor latency, demonstrating a level of sensitivity that exceeds conscious awareness. Our findings provide insights into latency effects on performance and perception, guiding the design of more responsive VR interaction systems.

As human-computer interaction (HCI) technology becomes more and more integrated into our daily life, increasing attention has been drawn towards the interaction experience in addition to HCI efficiency. In the present study, we conducted a survey to explore context-specific emotional experience in HCI. Four hundred participants were recruited to report the frequency of their emotional experiences on 44 fine-grained emotion items in six representative HCI scenarios. Compared with six matched human-human interaction (HHI) scenarios used as control, the HCI scenarios were in general more frequently associated with negative emotions, and less frequently associated with positive emotions, especially when computer served as a tool for communication with other people. Furthermore, the 44 emotional experience items in HCI were summarized as five factors, representing low-arousal focused, positively engaged, emotionally empathetic, high-arousal negative and frustratingly confused. Our study presents a comprehensive overview of context-specific emotional experience in human-computer interactions and provides a framework for emotion evaluation in HCI applications.

The view of embodied emotion believes that emotions are the emotions of the body. While emotion-specific patterns of self-reported bodily sensation have been previously reported, the physiological bodily representation across emotions remains to be addressed. The present study aimed to investigate the effectiveness of multi-site bodily electrodermal representations of emotions. A multi-channel electrodermal measurement device was designed to record electrodermal activities from nine body sites (neck, back, chest, bilateral abdomen, bilateral wrist, and bilateral ankle) from thirty-six college students (all male), while they were presented with a series of emotional pictures. Using the integral skin conductance response feature and a random forest classification method, the classification of high and low arousal levels achieved an average classification accuracy of 80.4±8.1%, and the classification of positive, neutral, and negative states reached an average classification accuracy of 76.4±10.2%. The classification models for arousal and valence were found to rely on distinct bodily representations. Meanwhile, the classification results of multi-site measurement were significantly better than single-site results. Our findings for the first time illustrate the bodily electrodermal representations of emotion and suggest the feasibility of affective computing using bodily electrodermal signals.

In this paper, we propose a novel self-distillation method for fake speech detection (FSD), which can significantly improve the performance of FSD without increasing the model complexity. For FSD, some fine-grained information is very important, such as spectrogram defects, mute segments, and so on, which are often perceived by shallow networks. However, shallow networks have much noise, which can not capture this very well. To address this problem, we propose using the deepest network instruct shallow network for enhancing shallow networks. Specifically, the networks of FSD are divided into several segments, the deepest network being used as the teacher model, and all shallow networks become multiple student models by adding classifiers. Meanwhile, the distillation path between the deepest network feature and shallow network features is used to reduce the feature difference. A series of experimental results on the ASVspoof 2019 LA and PA datasets show the effectiveness of the proposed method, with significant improvements compared to the baseline.

EEG signals have been reported to be informative and reliable for emotion recognition in recent years. However, the inter-subject variability of emotion-related EEG signals still poses a great challenge for the practical applications of EEG-based emotion recognition. Inspired by recent neuroscience studies on inter-subject correlation, we proposed a Contrastive Learning method for Inter-Subject Alignment (CLISA) to tackle the cross-subject emotion recognition problem. Contrastive learning was employed to minimize the inter-subject differences by maximizing the similarity in EEG signkal representations across subjects when they received the same emotional stimuli in contrast to different ones. Specifically, a convolutional neural network was applied to learn inter-subject aligned spatiotemporal representations from EEG time series in contrastive learning. The aligned representations were subsequently used to extract differential entropy features for emotion classification. CLISA achieved state-of-the-art cross-subject emotion recognition performance on our THU-EP dataset with 80 subjects and the publicly available SEED dataset with 15 subjects. It could generalize to unseen subjects or unseen emotional stimuli in testing. Furthermore, the spatiotemporal representations learned by CLISA could provide insights into the neural mechanisms of human emotion processing.

Emotion recognition from EEG signals has attracted much attention in affective computing. Recently, a novel dynamic graph convolutional neural network (DGCNN) model was proposed, which simultaneously optimized the network parameters and a weighted graph$G$characterizing the strength of functional relation between each pair of two electrodes in the EEG recording equipment. In this article, we propose a sparse DGCNN model which modifies DGCNN by imposing a sparseness constraint on$G$and improves the emotion recognition performance. Our work is based on an important observation: the tomography study reveals that different brain regions sampled by EEG electrodes may be related to different functions of the brain and then the functional relations among electrodes are possibly highly localized and sparse. However, introducing sparseness constraint into the graph$G$makes the loss function of sparse DGCNN non-differentiable at some singular points. To ensure that the training process of sparse DGCNN converges, we apply the forward-backward splitting method. To evaluate the performance of sparse DGCNN, we compare it with four representative recognition methods (SVM, DBN, GELM and DGCNN). In addition to comparing different recognition methods, our experiments also compare different features and spectral bands, including EEG features in time-frequency domain (DE, PSD, DASM, RASM, ASM and DCAU on different bands) extracted from four representative EEG datasets (SEED, DEAP, DREAMER, and CMEED). The results show that (1) sparse DGCNN has consistently better accuracy than representative methods and has a good scalability, and (2) DE, PSD, and ASM features on$\gamma$band convey most discriminative emotional information, and fusion of separate features and frequency bands can improve recognition performance.

Positive emotions are of great significance to people's daily life, such as human-computer/robot interaction. However, the structure of extensive positive emotions is not clear yet and effective standardized inducing materials containing as many positive emotional categories as possible are lacking. Thus, this article aims to establish a Chinese positive emotion database (CPED) to (1) effectively elicit positive emotion categories as many as possible, (2) provide both the subjective feelings of different positive emotions and a corresponding peripheral physiological database, and (3) explore the structure and framework of positive emotion categories. 42 video clips of 16 positive emotion categories were screened from 1000+ online clips. Then a total of 312 participants watched and rated these video clips during which GSR and PPG signals were recorded. 34 video clips that met hit rate and intensity standards were systemically clustered into four emotion categories (empathy, fun, creativity and esteem). Eventually, 22 film clips of these four major categories formed the CPED database. A total of 84 features from GSR and PPG signals were extracted and entered into RF, SVM, DBN and LSTM classifiers that serves as baseline classification methods. A classification accuracy of 44.66 percent for four major categories of positive emotions was achieved.

Compared with the widely studied negative emotions in which different classes are easy to distinguish, nowadays less attention is paid to the recognition of positive emotions that are not fully independent. In this article, we propose to recognize multiple continuous positive emotions that exhibit statistical dependencies using multi-target regression — by analyzing brain activities when an individual watches emotional film clips — and explore the neural representation of different positive emotions. Thirty-seven participants volunteered to participate in our study, in which their brain activities were recorded when watching five selected film clips (corresponding to five positive emotions: amusement, happiness, romance, tenderness and warmth). First, 150 well-known power features extracted from Electroencephalography (EEG) signals and 105 multimedia content analysis features were collected as the pool of candidate features. Second, based on the collected features, we propose to use a linear model (linear regression) and a nonlinear model (long short-term memory network, LSTM) to predict the percentage of five positive emotions. Then, percentage values were converted to ranking numbers and Kendall rank correlation coefficients were calculated. Our results showed that (1) ensemble of regressor chains (ERC) using LSTM as unit regressor obtained both the best regression results (with lowest RMSE = 8.325 and highest$\text{R }^{2} = 0.346$) and the best Kendall rank correlation coefficient (0.165) on EEG features merely, and (2) selective features from alpha frequency bands of EEG signals could represent different positive emotions. These results demonstrate the effectiveness of selective EEG features on recognizing different positive emotions.

The popularity of wearable physiological recording devices has opened up new possibilities for the assessment of personality traits in everyday life. Compared with traditional questionnaires or laboratory assessments, wearable device-based measurements can collect rich data about individual physiological activities in real-life situations without interfering with normal life, enabling a more comprehensive description of individual differences. The present study aimed to explore the assessment of individuals' Big-Five personality traits by physiological signals in daily life situations. A commercial bracelet was used to track the heart rate (HR) data from eighty college students (all male) enrolled in a special training program with a strictly-controlled daily schedule for ten consecutive working days. Their HR activities were divided into five daily situations (morning exercise, morning classes, afternoon classes, free time in the evening, and self-study situations) according to their daily schedule. Regression analyses with HR-based features in these five situations averaged across the ten days revealed significant cross-validated quantitative prediction correlations of 0.32 and 0.26 for the dimensions of Openness and Extraversion, with the prediction correlation trending significance for Conscientiousness and Neuroticism. Moreover, the multi-situation HR-based results were in general superior to those based on single-situation HR-based features, as well as those based on the multi-situation self-reported emotion ratings. Togetherour findings demonstrate the link between personality and daily HR measures using state-of-the-art commercial devices and could shed light on the development of Big-Five personality assessment based on daily multi-situation physiological measures.

The assessment of personality is crucial not only for scientific inquiries but also for real-world applications such as personnel selection. In this article, we propose and validate a novel implicit measure to predict an individual's levels in the Big Five personality traits from 5 minutes of electroencephalography (EEG) recordings. Participants viewed Chinese words with positive, negative, and neutral emotions. The multi-channel event-related potentials elicited by these emotional words were used to train a sparse regression model for personality prediction. Results from a large test sample of 196 participants indicated that the personality scores derived from the proposed measure reached significant correlations with a commonly used questionnaire (r = .50, .60, .49, .55, and .49 for agreeableness, conscientiousness, neuroticism, openness, and extraversion, respectively). The EEG-based personality scores showed good external validity as well, capable of predicting behavioral indices and psychological adjustment similar to self-reported scores. Besides, the EEG-based scores were relatively stable across time, as reflected by the test-retest reliability of .5 ∼ .7 for the five personality traits within a cohort of 33 participants 19-78 days later. These evaluations suggest that the proposed measure can serve as a viable alternative to conventional personality questionnaires in practice.

How to efficiently tag the emotional experience of multimedia contents is an important and challenging problem in the field of affective computing. This paper presents an EEG-based real-time emotion tagging approach, by extracting inter-brain features from a group of participants when they watch the same emotional video clips. First, the continuous subjective reports on both the arousal and valence dimensions of emotion were obtained by employing a three-round behavioral rating paradigm. Second, the inter-brain features were systematically explored in both spectral and temporal domain. Finally, regression analyses were performed to evaluate the effectiveness of inter-brain amplitude and phase features. The inter-brain amplitude feature showed significantly better prediction performance than the inter-brain phase feature, as well as another two conventional features (spectral power and inter-subject correlation). By combining the four types of features, regression values (R2) were obtained for the prediction of arousal (0.61 + 0.01) and valence (0.70 + 0.01), corresponding to prediction errors of 1.01 + 0.02 and 0.78 + 0.02 (unit on 9-point scales), respectively. The contributions of different electrodes and frequency bands were also analyzed. Our results show promising potentials of inter-brain EEG features in real-time emotion tagging applications.

Despite the popularity of multiplayer online battle arena (MOBA) games, academic research on MOBA is still very limited. The current study aimed to fill this gap by exploring the behavioral and neural correlates of expertise for the most popular MOBA game, League of Legends (LOL). Three groups of LOL players with different expertise levels were recruited, including professional players, background-matched trainees, and age-matched students with no systematic LOL trainings. A series of behavioral tests and questionnaires was used to evaluate their general cognitive skills and their LOL-specific abilities were extracted from the neural activities (Electroencephalographs (EEG)s and Electrocardiographs (ECG)s) recorded during LOL matches. Using the behavioral features, both the students and the trainees could be significantly separated from the professional players (trainees vs. professional players, 61.11%; students vs. professional players, 66.67%), whereas the students and the trainees cannot be distinguished. Using the neural features, all three groups could be well separated with higher classification accuracies (students vs. trainees: 88.24%; trainees vs. professional players, 93.33%; students vs. professional players, 93.75%). The most contributing behavioral and neural indices were revealed as well, including multiple-object tracking capability, mental concentration, visuospatial attention ability, etc. The authors’ results for the first time showed the possibility of recognizing MOBA expertise using both behavioral and neural measurements and provided a framework for evaluation, selection, and training of professional MOBA players.

Although accurate recognition of the idle state is essential for the application of brain-computer interfaces (BCIs) in real-world situations, it remains a challenging task due to the variability of the idle state. In this study, a novel algorithm was proposed for the idle state detection in a steady-state visual evoked potential (SSVEP)-based BCI. The proposed algorithm aims to solve the idle state detection problem by constructing a better model of the control states. For feature extraction, a maximum evoked response (MER) spatial filter was developed to extract neurophysiologically plausible SSVEP responses, by finding the combination of multi-channel electroencephalogram (EEG) signals that maximized the evoked responses while suppressing the unrelated background EEGs. The extracted SSVEP responses at the frequencies of both the attended and the unattended stimuli were then used to form feature vectors and a series of binary classifiers for recognition of each control state and the idle state were constructed. EEG data from nine subjects in a three-target SSVEP BCI experiment with a variety of idle state conditions were used to evaluate the proposed algorithm. Compared to the most popular canonical correlation analysis-based algorithm and the conventional power spectrum-based algorithm, the proposed algorithm outperformed them by achieving an offline control state classification accuracy of 88.0 ± 11.1% and idle state false positive rates (FPRs) ranging from 7.4 ± 5.6% to 14.2 ± 10.1%, depending on the specific idle state conditions. Moreover, the online simulation reported BCI performance close to practical use: 22.0 ± 2.9 out of the 24 control commands were correctly recognized and the FPRs achieved as low as approximately 0.5 event/min in the idle state conditions with eye open and 0.05 event/min in the idle state condition with eye closed. These results demonstrate the potential of the proposed algorithm for implementing practical SSVEP BCI systems.