The existing monocular methods face huge challenges in reconstructing credible details of non-visible areas in large pose images. Due to the fact that facial details are lost in non-visible areas of large pose images, existing methods lose basis when reconstructing details, resulting in unreliable results. Even if the generative model is used to repair the image first, the reconstruction process is very complex and costly. To this end, we propose an end-to-end and self-supervised RGB to depth method that equates small pose to large pose in UV space to obtain labels for non-visible areas. Then, we infer the depth values of non-visible areas and reconstruct a detailed 3D face. Finally, we render it into a face image and use it alongside the labels for self-supervised training of the network. During inference for large pose image, our method could reconstruct credible details of non-visible areas with a basis, rather than blindly. In addition, coarse and detailed reconstruction have mutually exclusive requirements for training images while coarse reconstruction is often not met, resulting in limited reconstruction accuracy. We propose a mutual exclusion elimination method to solve it, improving its accuracy. Extensive experiments demonstrate that our method could reconstruct precise 3D faces with credible details from large pose images. Our supplementary material is published on: https://github.com/lxy-nxu/ICASSP2025/tree/main

Despite significant advances in 3D human pose estimation from a single-view video, existing methods often struggle to produce reasonable human poses when the human is heavily occluded or blurred. To address this issue, we propose a Mixed Spectral and Attention Network (MSANet) that stacks spectral and attention blocks alternately. The attention block captures visual cues before and after occlusions or blurs, while the spectral block perceives subtle localized occlusions or blurs for robust 3D human pose estimation. Specifically, our attention block captures the global information of intra-frame joints and enhances the coherent representation of inter-frame joints, the spectral block complements the intra- and inter-frame local occlusions information which is difficult to capture by the attention block. In addition, we improve the regression head (IRH) to narrow the grained gap between joint-level feature extraction and frame-level pose regression for smooth regression. With better temporal consistency and subtle localized occlusion awareness, our MSANet outperforms previous state-of-the-art methods on the commonly used benchmarks Human3.6M and MPI-INF-3DHP. Moreover, MSANet demonstrates broad real world applicability, realizing occlusions and blurs robust and accurate 3D pose estimation. The Code will be made public.

Recently, great progress has been made in the field of video 3D human mesh reconstruction. Existing methods usually consider the correlation between features while ignoring essential feature learning, i.e. ignoring considering non-correlation between features, which results in learning a large amount of redundant and pseudo-correlated information, especially in complex scenarios. To address the above problem, we propose a multidimensional variance attention method that could learn non-correlation between features from multiple dimensions effectively. Specifically, we first design a variance attention network that filters out and weights non-correlation features by variance, the variance attention network can extract non-correlation features to approximate essential features and reduce redundancy. Furthermore, we design a multi-dimensional variance attention network that regards different dimensions as another kind of non-correlation, weights and fuses the selected features from time, channel and frequency domain dimensions to extract essential features. At the same time, the network extracts high-frequency information from the frequency domain by using a high-pass filter. These essential features and high-frequency information are fused to acquire the output that considers both correlation and non-correlation, which can conduct effective complementary learning. Extensive experiments on three benchmark datasets demonstrate the effectiveness of our method.

3D face alignment from monocular images in challenging scenarios such as large poses and occlusions presents a huge challenge. To overcome this challenge, we propose a Multi-granularity Perturbation Relearning Network (MPRN), utilizing relearning attention to capture crucial features. Specifically, MPRN employs an attention mechanism to highlight effective features and further conducts relearning for attention to refine its accuracy. However, in extreme scenarios, the loss of key 3D facial information hampers the effective functioning of relearning attention. To this end, we construct multi-granularity perturbation graphs to infer the missing key 3D facial information and correspondingly guide the multiple times learning of attention module using perturbation graphs at various granularities. By doing this, our MPRN could effectively capture crucial 3D facial features in extreme scenarios, thereby achieving precise 3D face alignment. Experiments on the AFLW 2000-3D and AFLW datasets demonstrate the effectiveness of our MPRN.

Speech-driven 3D facial animation aims to generate realistic and vivid 3D facial animations from speech.However, the scarcity of labeled data and the tendency of existing methods to treat this cross-modal mapping problem as a regression task can result in inadequate learning of discriminative features from the speech.This deficiency often leads to excessively smooth facial movements, particularly in lip movements.To address these issues and enhance the accuracy of lip generation while reducing reliance on labeled data, we propose CLTalk, a framework based on a contrastive learning strategy.This framework comprises three main parts: a temporal domain contrastive learning strategy that facilitates the learning of discriminative features from different audio frames, a correlation learning method that ensures consistency between the distribution of audio features and Mesh labels, and a mouth opening angle constraint method to further improve the accuracy of lip generation.Extensive experimental results on the challenging, widely evaluated datasets indicate the effectiveness of our method compared with the state of the arts.

Monocular image or video based 3D human pose estimation remains a very challenging task because of depth ambiguity and occluded joints. To relieve this limitation, we propose a Multiple Hybrid Extraction Network (MHENet), which obtains three different representations of pose hypotheses features by multiple hybrid extractors with different structures, and uses pose interaction and fusion to obtain accurate 3D pose. The Hybrid Extraction Module obtains three hypotheses features: base features correspond to structural information, diverse features correspond to detail information, and condensed features correspond to action information. Hypotheses Interaction Fusion Modul builds relationships across hypotheses feature to generate more accurate 3D poses. Extensive qualitative and quantitative experimental results on a large-scale publicly available dataset demonstrate that our approach achieves competitive performance compared to state-of-the-art methods. The code will be made publicly.