Personality primarily refers to the unique and stable way of a person’s thinking and behavior. A few studies have recently been conducted on personality recognition using physiological signals, most of which have used 2D emotional stimulus materials. Virtual reality (VR) has been utilized in many fields, and its superiority over 2D in emotion recognition has been proven. However, relevant research on VR scenes is lacking in the field of personality recognition. In this study, based on the psychological principle that emotional arousal can expose an individual’s personality, we attempt to explore the feasibility and effect of using electrocardiogram (ECG) signals in response to VR emotional stimuli for personality identification. For this purpose, a VR-2D emotion-induction experiment was conducted in which ECG signals were collected, and physiological datasets of emotional personalities were constructed through preprocessing and feature extraction. Statistical analysis of the emotion scale scores and ECG features of the participants showed that the VR group had a higher number of significantly correlated features. Meanwhile, VR- and 2D-based personality recognition models were constructed using machine learning algorithms. The results showed that the VR-based personality recognition model achieved better results for the four personality dimensions, with a maximum accuracy of 79.76%. These findings indicate that VR not only enhances the physiological correlation between emotion and personality but also improves the classification accuracy of personality recognition.

Stress, especially chronic stress, is a high risk factor of many physical and mental health problems. This work acquired 702 days of full-day ambulatory electrocardiogram (ECG) and Tri-axial acceleration (T-ACC) data from 104 healthy college students and realized chronic stress recognition through signal processing, statistical test and machine learning. We divided the 24 hours of a day into 153 time slots, and calculated 30 features from ECG and T-ACC data in each time slot. Statistical test of the above 30 features of the subjects with chronic stress and no chronic stress labels showed that chronic stress altered the autonomic nervous control of the heart not only in the daily activity time but also in the rest time at night, leading to smaller heart rate variability, faster heart rate and less complexity of the heartbeat rhythm. More specifically, the parasympathetic nervous activity at night was weakened by chronic stress. We expressed the ECG and T-ACC data of a day as a 30×153 data matrix, applied a four-layer fully connected neural network to classify the data of 702 days with chronic stress and no chronic stress labels, and obtained 88.17% chronic stress detection accuracy in the leave-one-subject-out cross test.

Olfactory-enhanced virtual reality (OVR) creates a complex and rich emotional experience, thus promoting a new generation of human-computer interaction experiences in real-world scenarios. However, with the rise of virtual reality (VR) as a mood induction procedure (MIP), few studies have incorporated olfactory stimuli into emotion induction in three-dimensional (3D) environments. Considering the differences in electroencephalography (EEG) dynamics between sensory stimuli, all previous two-dimensional (2D) and 3D emotional studies have been less effective in reality because they only use visual and audio senses. To overcome these limitations, we developed a novel EEG signal dataset based on OVR. We systematically analyzed the influence of olfactory stimuli on emotion induction in a VR environment from a neurophysiological perspective. Specifically, synchronous EEG signals were collected from 65 participants as they watched positive and negative videos in traditional VR and OVR. Their power spectral densities (PSDs) were then calculated to compare the differences in brain activation between their VR and OVR modes during the induction of positive and negative emotions, while their brain states were classified after feature selection. The results showed that olfactory stimuli enhanced EEG responses for positive emotions, but the opposite was true for negative emotions. Additionally, the recognition rate of brain emotional states was more than 90% under both positive and negative emotions, while the high-frequencyβandγbands could effectively distinguish VR and OVR modes. This study introduced the olfaction into the field of human-computer interaction, which could promote research on emotion induction and recognition in real-world environments.

The purpose of facial expression recognition is to recognize the corresponding emotions. However, people tend to hide their emotions by displaying facial expressions that differ from those evoked by emotions. These inconsistent facial expressions are referred to as masked facial expressions (MFEs). The automatic recognition of hidden emotions within an MFE using image data is challenging. In this study, we find distinctive movement patterns in the facial action units (AUs) of MFE sequences through a detailed analysis. Considering our findings, we propose handcrafted features called dynamic AU intensity features (DAIFs) to represent AU movement. Furthermore, we develop a decoupled AU transformer (DAUT) model for recognition, where the decoupled convolution operators ensure that the temporal information in the DAIF is not damaged. To further improve the recognition performance, we design self-supervised clip prediction for pretraining of DAUT. Experimental results demonstrate that our proposed method performs exceptionally well across all tasks in the MFE dataset, particularly improving accuracy by nearly double on the most challenging 36-class task. This suggests that leveraging temporal information from facial AU movements is a reliable and effective technique for recognizing MFEs.

Convolutional neural networks (CNNs) are commonly employed for image emotion recognition owing to their ability to extract local features; however, they have difficulty capturing the global representations of images. In contrast, self-attention modules in a visual transformer network can capture long-range dependencies as global features. Some studies have shown that an image's local and global features determine the emotions of the image and that some local regions can generate an emotional prioritization effect. Therefore, we proposed combining global self-attention features and a local multiscale features network (CGLF-Net) to recognize an image's emotion, extracting image features from global and local perspectives. Specifically, the cross-scale transformer network is employed instead of convolution operations in the global feature branch to enhance its model feature representation. In the local feature branch, the improved feature pyramid module is applied to extract features from different sensory fields, thereby combining semantic information with different scales. Furthermore, the local attention module based on class activation maps guides the network to focus on locally salient regions. In addition, using multibranch loss functions, local and global feature branches are combined to enhance the ability to capture a comprehensive set of features. Consequently, the proposed network achieves recognition accuracies of 75.61% and 65.01% on the FI-8 benchmark dataset and Emotion-6 benchmark dataset, respectively. These results show that the proposed CGLF-Net reliably address the difficulty of extracting global features using CNNs, representing the classification performance of the state-of-the-art.

Abstract Hyperspectral face recognition provides improved classification rates due to its abundant information in the face cubes of every subject in hyperspectral face databases. However, while offering excellent opportunities, it also brings new challenges, such as low signal‐to‐noise ratio, interband misalignment, and high data dimensionality. Based on these ad hoc problems, literature has already proposed some optimisation methods including dimensionality reduction, image denoising, and alignment to perform face recognition, yet lacking comprehensive evaluation. This paper proposes a novel hyperspectral face recognition algorithm that is based on spatial information fusion for feature extraction (histogram of local dynamic texture patterns) and collaborative representation classifier for classification. Meanwhile, the algorithm is applied to three popular hyperspectral face databases, Carnegie Mellon University (CMU)‐hyperspectral face database (HSFD), University of Western Australia (UWA)‐HSFD, and Hong Kong Polytechnic University (PolyU)‐HSFD databases. Experimental results demonstrate that CMU‐HSFD and UWA‐HSFD databases achieve very competitive classification results. PolyU‐HSFD database also achieves rather good classification rates. The best recognition results are 98.5% ± 0.95, 96.6% ± 0.98, and 94.0% ± 2.86 for CMU‐HSFD, UWA‐HSFD and PolyU‐HSFD, respectively. It demonstrates experimentally that this algorithm can be used to recognise faces. Moreover, we compared eight existing state‐of‐the‐art face recognition techniques with our proposed method in performing hyperspectral face recognition. In this research, we formulate hyperspectral face recognition as an image‐set classification problem and evaluate the performances compared with other kinds of algorithms. Comparisons with the eight existing hyperspectral face recognition techniques on three standard datasets show that the proposed algorithm outperforms most other state‐of‐the‐art algorithms, indicating that it is a promising approach for hyperspectral face recognition.

Social anxiety is a negative emotion which may impair the health of the heart and social functioning of an individual. This work analyzes the influence of social anxiety on the autonomic nerve control of the heart in two social exposure events: public speaking and thesis defending. In an experiment of public speaking, 59 human subjects were tested, and 11 conventional heartbeat measures and a heartbeat measure named the range of local Hurst exponents (RLHE) were evaluated for their capabilities to reveal the onset of social anxiety. Two-sample t-test between the baseline data and high anxiety data shows that social anxiety significantly reduces the complexity of the heartbeats. In an experiment of thesis defense, heartbeats data were acquired from nine graduate students. With the combination of three conventional features and the RLHE feature, a support vector machine classifier obtained true positive rate and true negative rate of 84.88 and 97.29 percent in the five-fold cross validation process of binary classification between high anxiety status and low anxiety status; the classifier also realized a generalization accuracy of 81.82 percent in detecting the high anxiety status in the thesis defense. A real-time anxiety monitoring system was established based on the above anxiety detecting method.

The detection of stress at early stages is beneficial to both individuals and communities. However, traditional stress detection methods that use physiological signals are contact-based and require sensors to be in contact with test subjects for measurement. In this paper, we present a method to detect psychological stress in a non-contact manner using a human physiological response. In particular, we utilize a hyperspectral imaging (HSI) technique to extract the tissue oxygen saturation (StO2) value as a physiological feature for stress detection. Our experimental results indicate that this new feature may be independent from perspiration and ambient temperature. Trier Social Stress Tests (TSSTs) on 21 volunteers demonstrated a significant difference $p\< 0.005$ and a large practical discrimination (d 1/4 1.37) between normalized baseline and stress StO2 levels. The accuracy for stress recognition from baseline using a binary classifier was 76.19 and 88.1 percent for the automatic and manual selections of the classifier threshold, respectively. These results suggest that the StO2 level could serve as a new modality to recognize stress at standoff distances.

Emotion recognition based on affective physiological changes is a pattern recognition problem, and selecting specific physiological signals is necessary and helpful to recognize the emotions. Fingertip blood oxygen saturation (OXY), galvanic skin response (GSR) and heart rate (HR) are acquired while amusement, anger, grief and fear of 101 subjects are individually elicited by films. The affective physiological changes in multi-subject GSR, the first derivative of GSR (FD_GSR) and HR are detected by the multi-variant correlation method. The correlation analysis reveals that multi-subject HR, GSR and FD_GSR fluctuations respectively have common intra-class affective patterns. In addition to the conventional features of HR and GSR, the affective HR, GSR and FD_GSR fluctuations are quantified by the local scaling dimension and applied as the affective features. The multi-subject affective database containing 477 cases is classified by a Random Forests classifier. An overall correct rate of 74 percent for quinary classification of amusement, anger, grief, fear and the baseline state are obtained.

TS (tabu search) is one of the meta-heuristic algorithms that can solve the combinatorial optimization problems, which is NP-hard such as TSPs (traveling salesman problems), satisfied. With the requirement of solving large-scale problems, we proposed a new parallel tabu search (PTS) approach, which was cooperated with genetic crossover operation for TSPs. In addition, a novel adaptive search strategy of intensification and diversification was proposed to improve the solution quality and efficiency in TS. Taking TSPs as sample, some simulation experiments were implemented. It was shown that our proposed PTS approach was feasible and effective.

Intensification strategy and diversification strategy are two highly important components of tabu search, which conflict each other in many cases. We proposed a novel forward search strategy that can encourage searching much thoroughly the neighborhood of elite solutions recorded, and can retain reasonable balance between intensification search and diversification search at the same time in order to get global optimal. Taking the classical combinatorial optimization problem TSP as samples, the conflict between intensification and diversification can be harmonized effectively by updating neighborhood, candidate and adjust tabu length automatically. Finally we give the experimental results.