Emotional talking head video generation aims to synthesize expressive portrait videos with accurate lip synchronization and emotional facial expressions. Current methods rely on simple emotional labels, leading to insufficient semantic information. Although high-level semantics enhance emotional expressiveness, using semantic guidance alone may compromise lip-sync precision. Furthermore, mainstream generation methods struggle to capture temporal dependencies across frames, resulting in degraded temporal coherence. Therefore, we propose an Emotion-Aware Diffusion model-based Network, called EAD-Net. We introduce SyncNet supervision and Temporal REPresentation Alignment (TREPA) to strengthen audio-visual alignment under multi-modal guidance. We further introduce a Spatio-Temporal Directional Attention (STDA) module to enhance fine-grained feature interactions via directional context aggregation, thereby improving audio-visual alignment. A Temporal Frame graph Reasoning Module (TFRM) is designed to explicitly model inter-frame dependencies, ensuring temporally coherent motion transitions without abrupt artifacts. To enhance emotional expressiveness, a large language model is employed to extract textual descriptions from real videos, serving as high-level semantic guidance. Experiments on the HDTF and MEAD datasets demonstrate that our method outperforms existing methods in terms of lip-sync accuracy, temporal consistency and emotional accuracy.

Current Audio-Visual Source Separation methods primarily adopt two design strategies. The first strategy involves fusing audio and visual features at the bottleneck layer of the encoder, followed by processing the fused features through the decoder. However, when there is a significant disparity between the two modalities, this approach may lead to the loss of critical information. The second strategy avoids direct fusion and instead relies on the decoder to handle the interaction between audio and visual features. Nonetheless, if the encoder fails to integrate information across modalities adequately, the decoder may be unable to effectively capture the complex relationships between them. To address these issues, this paper proposes a dynamic fusion method based on a gating mechanism that dynamically adjusts the modality fusion degree. This approach mitigates the limitations of solely relying on the decoder and facilitates efficient collaboration between audio and visual features. Additionally, an audio attention module is introduced to enhance the expressive capacity of audio features, thereby further improving model performance. Experimental results demonstrate that our method achieves significant performance improvements on two benchmark datasets, validating its effectiveness and advantages in Audio-Visual Source Separation tasks.

Vision-and-Language Navigation (VLN) is a challenging task where an agent must understand language instructions and navigate unfamiliar environments using visual cues. The agent must accurately locate the target based on visual information from the environment and complete tasks through interaction with the surroundings. Despite significant advancements in this field, two major limitations persist: (1) Many existing methods input complete language instructions directly into multi-layer Transformer networks without fully exploiting the detailed information within the instructions, thereby limiting the agent's language understanding capabilities during task execution; (2) Current approaches often overlook the modeling of object relationships across different modalities, failing to effectively utilize latent clues between objects, which affects the accuracy and robustness of navigation decisions. We propose a Dual Object Perception-Enhancement Network (DOPE) to address these issues to improve navigation performance. First, we design a Text Semantic Extraction (TSE) to extract relatively essential phrases from the text and input them into the Text Object Perception-Augmentation (TOPA) to fully leverage details such as objects and actions within the instructions. Second, we introduce an Image Object Perception-Augmentation (IOPA), which performs additional modeling of object information across different modalities, enabling the model to more effectively utilize latent clues between objects in images and text, enhancing decision-making accuracy. Extensive experiments on the R2R and REVERIE datasets validate the efficacy of the proposed approach.

Pronunciation Error Detection and Diagnosis (MDD) is a key technology in Computer-Assisted Pronunciation Training (CAPT) and Computer-Assisted Language Learning (CALL). Recently large language models (LLMs) have shown strong performance in multimodal tasks. This paper proposes a new MDD framework called S-TATLLM which combines the advantages of an incremental self-supervised model (based on a local-global feature extraction structure CGSL using multi-head self-attention and convolution) and large language models to build an end-to-end multimodal pronunciation error detection system. By introducing phoneme-level control information and a text-audio-text embedding approach the system guides the language model to focus on easily confused pronunciation errors thus improving detection performance. S-TATLLM achieves a recall of 99.44% an F1 score of 0.8281 and a diagnosis accuracy (DER) of 92.03% which are better than the wav2vec2-CTC method with 60.84% 0.6164 and 70.74% and the AEL method with 73.71% 0.7749 and 81.60%.

Diffusion models have demonstrated remarkable performance in speech synthesis, but typically require multi-step sampling, resulting in low inference efficiency. Recent studies address this issue by distilling diffusion models into consistency models, enabling efficient one-step generation. However, these approaches introduce additional training costs and rely heavily on the performance of pre-trained teacher models. In this paper, we propose ECTSpeech, a simple and effective one-step speech synthesis framework that, for the first time, incorporates the Easy Consistency Tuning (ECT) strategy into speech synthesis. By progressively tightening consistency constraints on a pre-trained diffusion model, ECTSpeech achieves high-quality one-step generation while significantly reducing training complexity. In addition, we design a multi-scale gate module (MSGate) to enhance the denoiser’s ability to fuse features at different scales. Experimental results on the LJSpeech dataset demonstrate that ECTSpeech achieves audio quality comparable to state-of-the-art methods under single-step sampling, while substantially reducing the model’s training cost and complexity.