Facial expression recognition (FER) has made significant progress over the past few years. But how to overcome the problem of high inter-class similarity and large intra-class difference in FER is still challenging. To address this problem, we propose a novel FER framework called AU-assisted Visual Transformer (AVT) by incorporating facial action units (AU) information into Visual Transformer, which mainly consists of three modules: Local Feature Extraction (LFE) module, Global Relationship Modeling (GRM) module and AU Fusion Module (AFM). Specifically, the LFE module aims to extract local facial expression features by using a deep convolutional neural network, the GRM module is a multi-layer Transformer encoder that captures the relation between local facial regions and obtains a global understanding of the face, and in particular, the AFM introduces fine-grained AU feature and fuses it with expression feature for final classification. Extensive experiments are conducted on RAF-DB and FERPlus datasets, and our AVT achieves competitive results compared to previous state-of-the-art methods, demonstrating the effectiveness of our approach.

As a typical psychological stress reaction, micro-expression (ME) is usually quickly leaked on a human face and can reveal the true feeling and emotional cognition. Therefore,automatic ME analysis (MEA) has essential applications in safety, clinical and other fields. However, the lack of adequate ME data has severely hindered MEA research. To overcome this dilemma and encouraged by current image generation techniques, this paper proposes a fine-grained ME generation method to enhance ME data in terms of data volume and diversity. Specifically, we first estimate non-linear ME motion using thin-plate spline transformation with a dense motion network. Then, the estimated ME motion transformations, including optical flow and occlusion masks, are sent to the generation network to synthesize the target facial micro-expression. In particular, we obtain the relative action units (AUs) of the source ME to the target face as a constraint to encourage the network to ignore expression-irrelevant movements, thereby generating fine-grained MEs. Through comparative experiments on CASME II, SMIC and SAMM datasets, we demonstrate the effectiveness and superiority of our method. Source code is provided in https://github.com/MEA-LAB-421/MEGC2022-Generation.