Audio-visual emotion recognition (AVER) is crucial in various AI applications, such as digital humans. However, due to the incomplete and heterogeneity in emotional expression between the audio and visual modalities, challenges remain in achieving effective emotion interaction and fusion. To address these issues, we propose a novel approach, Emotion-Aware Text Generation for Instructing Audio-Visual Emotion Recognition (ETG-AVER). This method generates emotion-Aware text from audio-visual modalities to explicitly capture emotional cues, and utilizes the generated text to guide emotional feature extraction and fusion, thereby improving the performance of AVER. Specifically, ETG-AVER consists of three key components: (1) Emotion-Aware Text Generation module, which first generates Emotion-Aware texts from audio and video modalities, compensating for the limitations of audio-visual modalities in emotional expression; (2) Text-Instruct Multimodal Feature Extraction and Fusion module, which uses the generated Emotion-Aware texts to precisely guide the extraction and fusion of audio-visual features, effectively enhancing cross-modal emotion interaction; (3) Hierarchical Loss Optimization module, which uses hierarchical loss functions to align emotional features across unimodal and multimodal sources, ensuring consistency in both semantic and representational levels, thereby improving cross-modal emotion recognition accuracy. Experimental results demonstrate that ETG-AVER significantly outperforms existing methods in terms of accuracy and fine-grained classification, achieving 60.78% WAR on MAFW and 78.36% WAR on DFEW, validating the effectiveness of our approach in refining emotion information transfer and recognition.

Abstract Self-supervised facial representation learning (SFRL) methods, especially contrastive learning (CL) methods, have been increasingly popular due to their ability to perform face understanding without heavily relying on large-scale well-annotated datasets. However, analytically, current CL-based SFRL methods still perform unsatisfactorily in learning facial representations due to their tendency to learn pose-insensitive features, resulting in the loss of some useful pose details. This could be due to the inappropriate positive/negative pair selection within CL. To conquer this challenge, we propose a Pose-disentangled Contrastive Facial Representation Learning (PCFRL) framework to enhance pose awareness for SFRL. We achieve this by explicitly disentangling the pose-aware features from non-pose face-aware features and introducing appropriate sample calibration schemes for better CL with the disentangled features. In PCFRL, we first devise a pose-disentangled decoder with a delicately designed orthogonalizing regulation to perform the disentanglement; therefore, the learning on the pose-aware and non-pose face-aware features would not affect each other. Then, we introduce a false-negative pair calibration module to overcome the issue that the two types of disentangled features may not share the same negative pairs for CL. Our calibration employs a novel neighborhood-cohesive pair alignment method to identify pose and face false-negative pairs, respectively, and further help calibrate them to appropriate positive pairs. Lastly, we devise two calibrated CL losses, namely calibrated pose-aware and face-aware CL losses, for adaptively learning the calibrated pairs more effectively, ultimately enhancing the learning with the disentangled features and providing robust facial representations for various downstream tasks. In the experiments, we perform linear evaluations on four challenging downstream facial tasks with SFRL using our method, including facial expression recognition, face recognition, facial action unit detection, and head pose estimation. Experimental results show that PCFRL outperforms existing state-of-the-art methods by a substantial margin, demonstrating the importance of improving pose awareness for SFRL. Our evaluation code and model will be available at https://github.com/fulaoze/CV/tree/main .

Self-supervised facial representation has recently attracted increasing attention due to its ability to perform face understanding without relying on large-scale annotated datasets heavily. However, analytically, current contrastive-based self-supervised learning (SSL) still performs unsatisfactorily for learning facial representation. More specifically, existing contrastive learning (CL) tends to learn pose-invariant features that cannot depict the pose details of faces, compromising the learning performance. To conquer the above limitation of CL, we propose a novel Pose-disentangled Contrastive Learning (PCL) method for general self-supervised facial representation. Our PCL first devises a pose-disentangled decoder (PDD) with a delicately designed orthogonalizing regulation, which disentangles the pose-related features from the face-aware features; therefore, pose-related and other pose-unrelated facial information could be performed in individual subnetworks and do not affect each other's training. Furthermore, we introduce a pose-related contrastive learning scheme that learns pose-related information based on data augmentation of the same image, which would deliver more effective face-aware representation for various downstream tasks. We conducted linear evaluation on four challenging downstream facial understanding tasks, i.e., facial expression recognition, face recognition, AU detection and head pose estimation. Experimental results demonstrate that PCL significantly outperforms cuttingedge SSL methods. Our Code is available at https://github.com/DreamMr/PCL.