The recognition of stress responses in health, resilience, and psychopathology holds significant scientific importance. The biological information carried by the facial tissue oxygen saturation (StO2) is a useful indicator for stress recognition. Although graph-based methods have achieved state-of-the-art (SOTA), there is room for improvement. This study proposes an automatic graph generation method for facial StO2, replacing manual methods and enhancing the information of graphs. To further enhance the capability of graphs’ representation, a dimensionality reduction strategy for nodes is proposed. The strategy is implemented based on an novel objective function that can increase the inter-class distance and reduce the intra-class distance. After obtaining highly representative graphs, a dual-stream network integrating GAT and GCN is designed to extract stress-related features from the graphs, ultimately achieving the SOTA recognition accuracy.

Micro-expression (ME) is a crucial cue to reveal hidden emotions and helps to diagnose mental illnesses such as depression. However, their rapid and subtle characteristics make them difficult to recognize, and relying on a single ME feature is insufficient to capture comprehensive information about hidden emotions. This paper proposes an inverted bottleneck fusion network (IBFNet), which combines ME and facial tissue oxygen saturation (StO2) for multimodal feature fusion to improve the recognition of hidden emotions. Specifically, IBFNet learns different representations of each modal feature, namely common and individual features. The modalities are then mapped to different subspaces for feature refinement. We design the recurrent cross-modal attention (RCA) module to explore commonalities across modalities, retaining modality-specific information through auxiliary branches to enhance diversity. Finally, the inverted bottleneck structure is used to fuse the common and individual feature of ME and StO2. Experimental results demonstrate a recognition accuracy of 90.47%, which surpasses the limitations of single-modality recognition.

The purpose of facial expression recognition is to recognize the corresponding emotions. However, people tend to hide their emotions by displaying facial expressions that differ from those evoked by emotions. These inconsistent facial expressions are referred to as masked facial expressions (MFEs). The automatic recognition of hidden emotions within an MFE using image data is challenging. In this study, we find distinctive movement patterns in the facial action units (AUs) of MFE sequences through a detailed analysis. Considering our findings, we propose handcrafted features called dynamic AU intensity features (DAIFs) to represent AU movement. Furthermore, we develop a decoupled AU transformer (DAUT) model for recognition, where the decoupled convolution operators ensure that the temporal information in the DAIF is not damaged. To further improve the recognition performance, we design self-supervised clip prediction for pretraining of DAUT. Experimental results demonstrate that our proposed method performs exceptionally well across all tasks in the MFE dataset, particularly improving accuracy by nearly double on the most challenging 36-class task. This suggests that leveraging temporal information from facial AU movements is a reliable and effective technique for recognizing MFEs.

Stress is an integral state that affects physical and mental health of individuals. Tissue Oxygen Saturation (StO2) is an emerging physiological signal and can reflect different stress states. Previous studies have simply extracted features of StO2 from specific regions of the face without exploring potential associations between them. In this paper, we analyze the facial Region of Interest (ROI) based on StO2, so as to explore the deep relationship between ROIs. Building upon this, we construct, for the first time, a StO2-based facial graph structure for stress recognition by using ROIs as nodes. In addition, we propose a new graph pooling method called FTPool which allows to measure node significance in terms of both node characteristics and graph topology. Finally, we further propose a dual-stream network (GCNet) combining GNN and CNN for the baseline-independent stress recognition. The experimental results show that the proposed GCNet can achieve state-of-the-art (SOTA) results with an accuracy of 78.57% on the original unbalanced database.

In this paper, we introduce a new biometric identity, facial tissue oxygen saturation (StO2). StO2 is an index of blood oxygen content in tissues and is related to blood vessel distribution pattern and metabolic rate. Experimental results show that classification accuracy can reach 83.33% in 42 participants with different stress states by using StO2 as the only input to the ResNet-50 model. We also proposed a module called StO2Net to eliminate the effects of stress on classification. The highest accuracy can reach up to 90.48% when the module is used. This pilot study shows that facial StO2 can be a promising biometric feature for identity recognition.

Facial expression, as a basic communication method, is an important way of emotion expression and cognition. Facial emotional expression impairment seriously affects interpersonal communication and social life. Micro-expressions (MEs) are involuntary and instant facial dynamics that occurs when the subject failed to suppress their genuine emotions, especially in high-stake situations. Psychological research has shown that MEs can reflect people’s true emotions, which is of great help to the treatment of Facial emotional expression impairment. ME spotting aims to locate the apex frame positions of MEs from long videos, which is the first step in ME analysis. Unlike previous researches that used binary classification or maximum feature difference for analysis, in this paper, we apply the idea of outlier detection to spot ME for the first time. MEs are unusual facial dynamics whose movement patterns diverge from others, so they can be regarded as outliers in the feature space of long videos. Our proposed method uses Gaussian model to estimate the probability density function and locates outliers by analyzing the statistical features of long videos to achieve ME spotting. This method was evaluated on CASME I, CASME II and SAMM datasets that only include spontaneous MEs in long videos. The results show that this method can efficiently locate apex frames of ME efficiently in long videos and also provide a new perspective for ME spotting.