The first Multimodal Emotion Recognition Challenge (MER 2023)1 was successfully held at ACM Multimedia. The challenge focuses on system robustness and consists of three distinct tracks: (1) MER-MULTI, where participants are required to recognize both discrete and dimensional emotions; (2) MER-NOISE, in which noise is added to test videos for modality robustness evaluation; (3) MER-SEMI, which provides a large amount of unlabeled samples for semi-supervised learning. In this paper, we introduce the motivation behind this challenge, describe the benchmark dataset, and provide some statistics about participants. To continue using this dataset after MER 2023, please sign a new End User License Agreement2 and send it to our official email address3. We believe this high-quality dataset can become a new benchmark in multimodal emotion recognition, especially for the Chinese research community.

Micro-expression (ME) is a significant non-verbal communication clue that reveals one person's genuine emotional state. The development of micro-expression analysis (MEA) has just gained attention in the last decade. However, the small sample size problem constrains the use of deep learning on MEA. Besides, ME samples distribute in six different databases, leading to database bias. Moreover, the ME database development is complicated. In this article, we introduce a large-scale spontaneous ME database: CAS(ME)3. The contribution of this article is summarized as follows: (1) CAS(ME)3offers around 80 hours of videos with over 8,000,000 frames, including manually labeled 1,109 MEs and 3,490 macro-expressions. Such a large sample size allows effective MEA method validation while avoiding database bias. (2) Inspired by psychological experiments, CAS(ME)3provides the depth information as an additional modality unprecedentedly, contributing to multi-modal MEA. (3) For the first time, CAS(ME)3elicits ME with high ecological validity using the mock crime paradigm, along with physiological and voice signals, contributing to practical MEA. (4) Besides, CAS(ME)3provides 1,508 unlabeled videos with more than 4,000,000 frames, i.e., a data platform for unsupervised MEA methods. (5) Finally, we demonstrate the effectiveness of depth information by the proposed depth flow algorithm and RGB-D information.

Micro-expression, which is generated by facial muscle movements, could be a crucial cue for deception detection. In the existed research investigating the relationship between facial muscles and deception detection, researchers have focused almost exclusively on two muscles, i.e., zygomaticus and corrugator supercilii, based on the theoretical basis that they are highly associated with positive and negative expressions. However, the aim of this study is to demonstrate the direct relationship between facial muscle movements and deception detection. Addressing this issue, this paper proposes an experimental paradigm with high ecological validity that uses electromyography (EMG) signals to precisely examine the role of facial muscle movements in deception detection. Moreover, we propose a vector-based sequential forward selection (VSFS) algorithm to identify the muscle (or muscle combination) most closely associated with lying. Based on our proposed approach, the importance of seven selected facial muscles was explored by comparing the corresponding facial EMG (fEMG) between truth and lying conditions. First, the present study found that the zygomaticus and corrugator supercilii could play important roles in deception detection, and our findings are consistent with existed research. Second, the experiment result verified that the muscles related to deception detection were consistent with those with higher frequency occurring in micro-expression. Moreover, the present study provides a theoretical basis that intelligent micro-expressions analysis could improve the lie detection performance by focusing on the area of the forehead, eyebrows, and cheeks.

Designing a tactile image for blind people is a significant challenge due to the difficulty of recognizing objects on a 2D line drawing image by touch compared to vision. In this paper, we proposed ”tactile accuracy”, a new criterion to evaluate the performance of recognizing 242 raised line images of common objects for 30 subjects (10 blindfolded sighted subjects, 10 congenitally blind subjects, and 10 late blind subjects), instead of the conventional ”naming accuracy” used in the visual image recognition tasks. We used multi-level evaluation criteria including ”tactile accuracy” to systematically analyze the design factors in tactile images. The results showed that using multi-level evaluate criteria could help unveil the tactile cognitive preferences of different types of subjects for personalized learning. Moreover, we reported important design factors that affect tactile image recognition, thus providing guidelines on the design of tactile images.

The effects of gaze direction on the perception of facial expressions have been getting more and more attention from psychologists, but their implication for facial expression recognition in computer science is neglected. In the present psychological study, we investigated the effects of gaze direction on the perception of facial expressions, and focused on the role of social dominance of facial expressions. Experiments 1 and 2 examined the effects of gaze on evaluations of dominance, pleasure and arousal of neutral, joy, angry and disgust facial expressions, using Self-Assessment Manikin (SAM). Experiment 3 examined the influence of gaze direction on the efficiency of facial expression perception. The results indicated that (a) gaze direction influenced the dominance of neutral and angry facial expressions, while it did not influence the dominance of happy or disgust facial expressions; and (b) averted gaze facilitated the processing of angry faces. The results of three experiments suggested that for threatening facial expression (such as anger), social dominance might be the mediator in the effect of gaze on perception of facial expressions. The implication for facial expression recognition of computer science was discussed.