The speech-driven facial animation technology is generally categorized into two main types: 3D and 2D talking face. Both of these have garnered considerable research attention in recent years. However, to our knowledge, the research into 3D talking face has not progressed as deeply as that of 2D talking face, particularly in terms of lip-sync and perceptual mouth movements. The lip-sync necessitates an impeccable synchronization between mouth motion and speech audio. The speech perception derived from the perceptual mouth movements should resemble that of the driving audio. To mind the gap between the two sub-fields, we propose Learn2Talk, a learning framework that enhances 3D talking face network by integrating two key insights from the field of 2D talking face. First, drawing inspiration from the audio-video sync network, we develop a 3D sync-lip expert model for the pursuit of lip-sync between audio and 3D facial motions. Second, we utilize a teacher model, carefully chosen from among 2D talking face methods, to guide the training of the audio-to-3D motions regression network, thereby increasing the accuracy of 3D vertex movements. Extensive experiments demonstrate the superiority of our proposed framework over state-of-the-art methods in terms of lip-sync, vertex accuracy and perceptual movements. Finally, we showcase two applications of our framework: audio-visual speech recognition and speech-driven 3D Gaussian Splatting-based avatar animation.

In human-centric content generation, the pre-trained text-to-image models struggle to produce user-wanted por-trait images, which retain the identity of individuals while exhibiting diverse expressions. This paper introduces our efforts towards personalized face generation. To this end, we propose a novel multi-modal face generation frame-work, capable of simultaneous identity-expression control and more fine-grained expression synthesis. Our expression control is so sophisticated that it can be specialized by the fine-grained emotional vocabulary. We devise a novel dif-fusion model that can undertake the task of simultaneously face swapping and reenactment. Due to the entanglement of identity and expression, separately and precisely control-ling them within one framework is a nontrivial task, thus has not been explored yet. To overcome this, we propose sev-eral innovative designs in the conditional diffusion model, including balancing identity and expression encoder, improved midpoint sampling, and explicitly background con-ditioning. Extensive experiments have demonstrated the controllability and scalability of the proposed framework, in comparison with state-of-the-art text-to-image, face swap-ping, and face reenactment methods.

Recent facial texture generation methods prefer to use deep networks to synthesize image content and then fill in the UV map, thus generating a compelling full texture from a single image. Nevertheless, the synthesized texture UV map usually comes from a space constructed by the training data or the 2D face generator, which limits the methods' generalization ability for in-the-wild input images. Consequently, their facial details, structures and identity may not be consistent with the input. In this paper, we address this issue by proposing a style transfer-based facial texture refinement method named FaceRefiner. FaceRefiner treats the 3D sampled texture asstyleand the output of a texture generation method ascontent. The photo-realistic style is then expected to be transferred from the style image to the content image. Different from current style transfer methods that only transfer high and middle level information to the result, our style transfer method integrates differentiable rendering to also transfer low level (or pixel level) information in the visible face regions. The main benefit of suchmulti-levelinformation transfer is that, the details, structures and semantics in the input can thus be well preserved. The extensive experiments on Multi-PIE, CelebA and FFHQ datasets demonstrate that our refinement method can improve the texture quality and the face identity preserving ability, compared with state-of-the-arts.

In this paper, we tackle a special face completion task, facial displacement completion, which can offer a key component for many single image 3D face reconstruction systems. To produce a detailed 3D face with ear-to-ear complete displacement UV map, we propose a novel Displacement Completion method based on Transformer (DCT). Current transformer based image inpainting methods usually follow a two-stage scheme, which firstly recovers the masked pixels in low resolution with transformer, and then replenishes the inpainting result in high resolution with GAN. Although these methods have achieved great success, they suffer from information loss from two aspects when applied in face completion: 1) The downsampling operation makes transformer only produce a coarse appearance prior for GAN, incurring middle and low level information loss. 2) Some meaningful facial semantics can be well captured by transformer and further benefit the completion, but it's has not yet been explored. Motivated by the above considerations, we come up with three key designs in the proposed DCT: PCA tokenization, BERT-style learning, and style modulation. Firstly, we use PCA tokenization to replace the downsampling in transformer to preserve more meaningful structures. Secondly, we make transformer simulate the two tasks in BERT, Masked Language Model (MLM) and Next Sentence Prediction (NSP), for both masked pixels and facial attributes recovery. Thirdly, we encode the outcome of transformer as the latent code to guide an image translation network in the StyleGAN2 modulation way. Experments on both FaceScape dataset and in-the-wild data demonstrate DCT's better performance compared with other transformer based or GAN based completion methods.

Although remarkable progress has been made in recent years, current multi-exposure image fusion (MEF) research is still bounded by the lack of real ground truth, objective evaluation function, and robust fusion strategy. In this paper, we study the MEF problem from a new perspective. We don’t utilize any synthesized ground truth, design any loss function, or develop any fusion strategy. Our proposed method EMEF takes advantage of the wisdom of multiple imperfect MEF contributors including both conventional and deep learning-based methods. Specifically, EMEF consists of two main stages: pre-train an imitator network and tune the imitator in the runtime. In the first stage, we make a unified network imitate different MEF targets in a style modulation way. In the second stage, we tune the imitator network by optimizing the style code, in order to find an optimal fusion result for each input pair. In the experiment, we construct EMEF from four state-of-the-art MEF methods and then make comparisons with the individuals and several other competitive methods on the latest released MEF benchmark dataset. The promising experimental results demonstrate that our ensemble framework can “get the best of all worlds”. The code is available at https://github.com/medalwill/EMEF.