Face-based Voice Conversion (FVC) is a novel task that leverages facial images to generate the target speaker's voice style. Previous work has two shortcomings: (1) suffering from obtaining facial embeddings that are well-aligned with the speaker's voice identity information, and (2) inadequacy in decoupling content and speaker identity information from the audio input. To address these issues, we present a novel FVC method, Identity-Disentanglement Face-based Voice Conversion (ID-FaceVC), which overcomes the above two limitations. More precisely, we propose an Identity-Aware Query-based Contrastive Learning (IAQ-CL) module to extract speaker-specific facial features, and a Mutual Information-based Dual Decoupling (MIDD) module to purify content features from audio, ensuring clear and high-quality voice conversion. Besides, unlike prior works, our method can accept either audio or text inputs, offering controllable speech generation with adjustable emotional tone and speed. Extensive experiments demonstrate that ID-FaceVC achieves state-of-the-art performance across various metrics, with qualitative and user study results confirming its effectiveness in naturalness, similarity, and diversity.

Text-to-speech (TTS) has advanced from generating natural-sounding speech to enabling fine-grained control over attributes like emotion, timbre, and style.Driven by rising industrial demand and breakthroughs in deep learning, e.g., diffusion and large language models (LLMs), controllable TTS has become a rapidly growing research area.This survey provides the first comprehensive review of controllable TTS methods, from traditional control techniques to emerging approaches using natural language prompts.We categorize model architectures, control strategies, and feature representations, while also summarizing challenges, datasets, and evaluations in controllable TTS.This survey aims to guide researchers and practitioners by offering a clear taxonomy and highlighting future directions in this fast-evolving field.One can visit https://github.com/imxtx/ awesome-controllabe-speech-synthesis for a comprehensive paper list and updates.

Pre-Trained vision-language models, like CLIP, make few-shot action recognition possible via text prompt. However, teaching scenarios are complex and CLIP has difficulties in understanding semantic domain concepts of teaching. Meanwhile, due to the absence of spatial-temporal information in CLIP’s pre-training, we can’t directly apply CLIP to video tasks. To address above challenges, we develop a novel Vision-Text Dual-Tower framework for Few-Shot Teaching Action Recognition in videos. Specifically, domain adapter is proposed to store domain knowledge via a cache dictionary, and prompt learner is proposed to automatically construct prompt templates suitable for teaching scenarios. Besides, to obtain spatial-temporal features, we firstly segment teachers from the video frames, then extract features via a self-attention mechanism. Furthermore, to get credible results in real teaching scenarios, we build and release TBAD dataset, which covers eight classes of teaching actions. Experimental results on TBAD dataset show that our approach achieves a 15.6% improvement in zero-shot accuracy over the original CLIP model and a 40.0% enhancement in few-shot(eight-shot) accuracy relative to the zero-shot CLIP model. The persuasive results underscore its ability to recognize video actions with limited samples, particularly in challenging teaching scenarios. Codes and datasets are available at: https://github.com/gexiaoxiao7/TBA_CLIP_Net.