The key to facial expression recognition is to learn discriminative spatial-temporal representations that embed facial expression dynamics. Previous studies predominantly rely on pre-trained Convolutional Neural Networks (CNNs) to learn facial appearance representations, overlooking the relationships between facial regions. To address this issue, this paper presents an Appearance- and Relation-aware Parallel Graph attention fusion Network (ARPGNet) to learn mutually enhanced spatial-temporal representations of appearance and relation information. Specifically, we construct a facial region relation graph and leverage the graph attention mechanism to model the relationships between facial regions. The resulting relational representation sequences, along with CNN-based appearance representation sequences, are then fed into a parallel graph attention fusion module for mutual interaction and enhancement. This module simultaneously explores the complementarity between different representation sequences and the temporal dynamics within each sequence. Experimental results on three facial expression recognition datasets demonstrate that the proposed ARPGNet outperforms or is comparable to state-of-the-art methods.

Dynamic facial action recognition is key to understanding human emotions and behaviors, yet estimating facial action units (AUs) intensities in videos is difficult due to subtle muscle motions and complex spatial-temporal dependencies. Existing methods often use fixed or coarse graphs, limiting the ability to capture intricate AU relations and long-range dynamics. This paper presents a hierarchical framework CDAU to effectively capture Continuous Dynamics for AU intensity estimation task. Our approach dynamically constructs multiscale graphs for fine-grained spatiotemporal AU interactions and adaptively fusing information across levels. A bidirectional state-space module further captures long-range temporal dependencies. Extensive experiments on FEAFA and DISFA show that CD-AU outperforms existing methods in both ICC and MAE metrics, validating its generalization capability and stability across subjects and expressions.

Cross-Modal alignment is crucial for multimodal representation fusion due to the inherent heterogeneity between modalities. While Transformer-Based methods have shown promising results in modeling inter-modal relationships, their quadratic computational complexity limits their applicability to long-sequence or large-scale data. Although recent Mamba-Based approaches achieve linear complexity, their sequential scanning mechanism poses fundamental challenges in comprehensively modeling cross-modal relationships. To address this limitation, we propose Align-Mamba, an efficient and effective method for multimodal fusion. Specifically, grounded in Optimal Transport, we introduce a local cross-modal alignment module that explicitly learns token-level correspondences between different modalities. Moreover, we propose a global cross-modal alignment loss based on Maximum Mean Discrepancy to implicitly enforce the consistency between different modal distributions. Finally, the unimodal representations after local and global alignment are passed to the Mamba backbone for further cross-modal interaction and multimodal fusion. Extensive experiments on complete and incomplete multimodal fusion tasks demonstrate the effectiveness and efficiency of the proposed method. For instance, on the CMU-MOSI dataset, AlignMamba improves classification accuracy by 0.9%, reduces GPU memory usage by 20.3%, and decreases inference time by 83.3%.

Human emotions vary over time, which can be vividly described as a stream of emotions. Observing the emotion stream in daily life provides valuable insights into an individual's mental state. However, existing research in emotion understanding has mainly focused on classification tasks, assigning an emotion category to a well-trimmed segment or each frame within a continuous signal. In contrast, the task of temporal emotion detection, which involveslocatingthe boundaries of emotion segments andrecognizingtheir categories in untrimmed signals, has not been fully explored. To advance research in this area, this paper introduces an in-the-wild Audio-Visual Emotion Stream (AVES) dataset, which is reliably annotated with the time boundaries and emotion category for each emotion segment in the videos. Thus, AVES can serve as a solid benchmark for temporal emotion detection tasks. Moreover, considering the flexible boundaries and varying durations of emotion segments, we propose a Boundary Combination Network (BoCoNet) for temporal emotion detection, which leverages short-term temporal context information to first predict the boundaries of emotion segments and then locate the entire emotion segments. Extensive experiments conducted on various representative unimodal and multimodal representations demonstrate that BoCoNet achieves state-of-the-art results. The AVES dataset will be released to the research community. We expect that this paper can advance the research on emotion stream and temporal emotion detection.

This paper presents our solution for the Semi-Supervised Multimodal Emotion Recognition Challenge (MER2023-SEMI), addressing the issue of limited annotated data in emotion recognition. Recently, the self-training-based Semi-Supervised Learning~(SSL) method has demonstrated its effectiveness in various tasks, including emotion recognition. However, previous studies focused on reducing the confirmation bias of data without adequately considering the issue of data imbalance, which is of great importance in emotion recognition. Additionally, previous methods have primarily focused on unimodal tasks and have not considered the inherent multimodal information in emotion recognition tasks. We propose a simple yet effective semi-supervised multimodal emotion recognition method to address the above issues. We assume that the pseudo-labeled samples with consistent results across unimodal and multimodal classifiers have a more negligible confirmation bias. Based on this assumption, we suggest using a class-balanced strategy to select top-k high-confidence pseudo-labeled samples from each class. The proposed method is validated to be effective on the MER2023-SEMI Grand Challenge, with the weighted F1 score reaching 88.53% on the test set.

Significant progress has been made in graph representation learning in recent years. However, most of these methods model spatial relationships via predefined graphs or decouple spatial-temporal representations, which limits the generalization and effectiveness of the model. To address these issues, we introduce an adaptive multi-graph contrastive transformer (AMGCT) for general spatial-temporal graph representation learning. Specifically, we first propose adaptive multi-graph contrastive learning (AMGCL). Without any expert knowledge, AMGCL can gradually generate adaptive spatial graphs with different topologies to learn spatial representations from different views. Cross-graph contrastive learning further explores potential correlations between different views, making each view's features more discriminative. In addition, to avoid insufficient interaction caused by decoupling spatial-temporal information in existing methods, we design a coupled graph transformer (CGT) to consider spatial relationships at each stage of temporal modeling, explore complementary information between spatial and temporal domains, and obtain more compact spatial-temporal representations. Experimental results on two different spatial-temporal graph datasets and tasks demonstrate that the proposed method achieves excellent performance.

Craniofacial reconstruction is to get a visual outlook of an individual from its skull. It is an important technology in both forensic medicine and archeology. This paper proposes a novel craniofacial reconstruction method based on least square support vector regression (LSSVR), which has the flexibility for uncovering nonlinear relationships between variables and is easy to solve. We firstly build statistical shape models for skulls and face skins respectively, and then train the LSSVR model in the shape parameter spaces. Given an unknown skull, we project it to the skull shape parameter space, and use the LSSVR model to reconstruct the corresponding face skin. Cross validation is used for parameter selection in LSSVR. Experiments are given on a data set including 150 training pairs of skull and skin samples and 58 testing ones. Comparisons with ridge regression and partial least square regression show that our method can reconstruct the craniofacial effectively and accurately.

Skull identification is an important subject for research in forensic medicine. Current research can be divided into two categories: 1) craniofacial superimposition and 2) craniofacial reconstruction. Both categories rely essentially on the accurate extraction and representation of the intrinsic relationship between the skull and face in terms of the morphology, which still remain unsolved. They have high uncertainty and a low identification capability. This paper proposes a novel skull identification method that matches an unknown skull with enrolled 3D faces, in which the mapping between the skull and face is obtained using canonical correlation analysis. Unlike existing techniques, this method needs no accurate relationship between the skull and face, and measures only the correlation between them. In order to measure the correlation more reliably and improve the identification capability of the correlation analysis model, a region fusion strategy is adopted. Experimental results validate the proposed method, and show that the region-based method can significantly boost the matching accuracy. The correct identification rate reaches 94% when using a CT data set. This paper can provide a theory support for research on craniofacial superimposition and craniofacial reconstruction.