Talking face generation is a technique that synthesizes realistic, speech-synchronized facial animations from static images driven by audio or textual input. In recent years, notable advancements have been achieved in the generation of realistic lip-synchronized talking face videos. Although these methods have demonstrated highly realistic results in lip-sync generation, they rarely focus on the expression of emotions, which is essential for achieving emotionally rich and lifelike talking face videos. In this article, we propose a novel emotionally controllable audio-driven talking face generation framework, termed ECATFG. Specifically, the proposed ECATFG mainly consists of three modules: template video generator, landmark generator, and rendering module. Firstly, we integrate a customized motion transfer module into StyleGAN to generate template videos that encapsulate the target emotions and head movements. Subsequently, a Mamba-based landmark generator is proposed to accurately model the correlation between input audio and facial landmarks. Afterwards, in the rendering module, we employ a framework that combines both AdaIN and SPADE layers to transform audio-predicted landmarks into highly realistic images, guided by reference images and facial contour landmarks. Finally, comprehensive experiments validate the effectiveness of ECATFG, demonstrating its capability to generate high-quality talking face videos that not only achieve exceptional lip-synchronization accuracy but also effectively convey a wide range of emotions.

Automatic micro-expression recognition (MER) has essential applications in the psychological field. Graph-based models, due to their advantages in analyzing regionalized faces, have become a powerful method for MER. However, how to construct a graph from ME videos remains to be studied. To solve this problem, we design an adaptive graph attention network with temporal fusion to model the dynamic relationships between facial regions of interest (ROIs). Specifically, we first propose adaptive graph attention to establish learnable spatial graphs from ME videos. Then, we adopt an optical-flow-based feature as the suitable input for the graph network. In addition, an implicit semantic data augmentation algorithm is employed and improved as a data-driven weighted loss for better performance. Extensive experiments on SMIC-HS, CASME II and SAMM datasets have demonstrated the effectiveness of the proposed method, and it achieves to be the first graph-based model where UF1 and UAR both exceed 0.90 for 3-classes MER on CASME II. Code will be available at https://github.com/MEA-LAB-421/ICME2023-Recognition.

Facial expression recognition (FER) has made significant progress over the past few years. But how to overcome the problem of high inter-class similarity and large intra-class difference in FER is still challenging. To address this problem, we propose a novel FER framework called AU-assisted Visual Transformer (AVT) by incorporating facial action units (AU) information into Visual Transformer, which mainly consists of three modules: Local Feature Extraction (LFE) module, Global Relationship Modeling (GRM) module and AU Fusion Module (AFM). Specifically, the LFE module aims to extract local facial expression features by using a deep convolutional neural network, the GRM module is a multi-layer Transformer encoder that captures the relation between local facial regions and obtains a global understanding of the face, and in particular, the AFM introduces fine-grained AU feature and fuses it with expression feature for final classification. Extensive experiments are conducted on RAF-DB and FERPlus datasets, and our AVT achieves competitive results compared to previous state-of-the-art methods, demonstrating the effectiveness of our approach.

As a typical psychological stress reaction, micro-expression (ME) is usually quickly leaked on a human face and can reveal the true feeling and emotional cognition. Therefore,automatic ME analysis (MEA) has essential applications in safety, clinical and other fields. However, the lack of adequate ME data has severely hindered MEA research. To overcome this dilemma and encouraged by current image generation techniques, this paper proposes a fine-grained ME generation method to enhance ME data in terms of data volume and diversity. Specifically, we first estimate non-linear ME motion using thin-plate spline transformation with a dense motion network. Then, the estimated ME motion transformations, including optical flow and occlusion masks, are sent to the generation network to synthesize the target facial micro-expression. In particular, we obtain the relative action units (AUs) of the source ME to the target face as a constraint to encourage the network to ignore expression-irrelevant movements, thereby generating fine-grained MEs. Through comparative experiments on CASME II, SMIC and SAMM datasets, we demonstrate the effectiveness and superiority of our method. Source code is provided in https://github.com/MEA-LAB-421/MEGC2022-Generation.

Micro-expressions (MEs) are significant and effective clues to reveal the true feelings and emotions of human beings, and thus MEs analysis is widely used in different fields such as medical diagnosis, interrogation and security. However, it is extremely difficult to elicit and label MEs, resulting in a lack of sufficient MEs data for MEs analysis. To address this challenge and inspired by the current face generation technology, in this paper we introduce Generative Adversarial Network based on fine-grained Action Units (AUs) modulation to generate MEs sequence (FAMGAN). Specifically, after comprehensively analyzing the factors that lead to inaccurate AU values detection, we performed fine-grained AUs modulation, which includes carefully eliminating the various noises and dealing with the asymmetry of AUs intensity. Additionally, we incorporate super-resolution into our model to enhance the quality of the generated images. Through experiments, we show that the system achieves very competitive results on the Micro-Expression Grand Challenge (MEGC2021).