We introduce a novel framework for one-shot audio-driven talking head generation. Unlike prior works that require additional driving sources for controlled synthesis in a deterministic manner, we instead sample all holistic lip-irrelevant facial motions (i.e. pose, expression, blink, gaze, etc.) to semantically match the input audio while still maintaining both the photo-realism of audio-lip synchronization and overall naturalness. This is achieved by our newly proposed audio-to-visual diffusion prior trained on top of the mapping between audio and non-lip representations. Thanks to the probabilistic nature of the diffusion prior, one big advantage of our framework is it can synthesize diverse facial motion sequences given the same audio clip, which is quite user-friendly for many real applications. Through comprehensive evaluations of public benchmarks, we conclude that (1) our diffusion prior outperforms auto-regressive prior significantly on all the concerned metrics; (2) our overall system is competitive with prior works in terms of audio-lip synchronization but can effectively sample rich and natural-looking lip-irrelevant facial motions while still semantically harmonized with the audio input.

Digital color reproduction[1][2] is a technique that can change the color appearance of objects shown in an image or video. The technique is utilized in various aspects and scenarios including projection-based appearances editing[3], medical imaging[4] and discrimination improvement for dichromats or anomalous trichromats[5][6]. However, traditional object color reproduction has high computational complexity due to color space transform, temporal color consistency handling and object boundary decision, which makes real-time processing for videos impossible without additional computing power such as GPGPU acceleration[7]. In this paper, a real-time high quality fuzzy rule-based(FRB) color reproduction system for clothing with Intel®RealSenseTMSDK and 3D Camera is proposed. By virtue of the SDK and 3D camera, depth image can be used to accelerate the user foreground extraction and clothing boundary decision, which not only provides a high quality reproduction result with better boundary fidelity, but also speeds up the computation to achieve real-time processing capability.