Text-to-speech (TTS) has seen significant advancements in high-quality, expressive speech synthesis. However, achieving diverse and natural prosody in synthesized speech remains challenging. In this paper, we propose ProsodyFlow, an end-to-end TTS model that integrates large self-supervised speech models and conditional flow matching to model prosodic features effectively. Our approach involves using a speech LLM to extract acoustic features, mapping these features into a prosody latent space, and then employing conditional flow matching to generate prosodic vectors conditioned on the input text. Experiments on the LJSpeech dataset show that ProsodyFlow improves synthesis quality and efficiency compared to existing models, achieving more prosodic and expressive speech synthesizing.

Video outpainting aims to extend the content of a video beyond its original spatial boundaries. Existing methods tend to condition the generation process on a single frame or caption, failing to address the challenge in long videos with multiple video clips. To address this, we extend the diffusion-based image inpainting to a 3D diffusion model with a motion module for video outpainting. Then, we design four conditioning schemes to seek the appropriate condition strategy: Frame-wise context, Frame-wise Adapter, Text, and Free Condition. Additionally, to enhance the continuity in long video generation, we propose a novel Motion Momentum Update (MMU) method based on a training-free technique. Experimental results demonstrate that our proposed Free Condition strategy, termed Free-Outpainter, achieves state-of-the-art performance in video outpainting tasks. Ablation studies also show that the condition-free training paradigm is the most effective in avoiding inaccurate external information and fully utilizing inherent video data. Benefiting from the Free Condition strategy, our pipeline reduces the inference time to 5 seconds for 512x512x16 frames with only 10 GB of GPU memory. More results can be found on our demo page https://lilyn3125.github.io/FreeOutpainting/.

With the rapid development of speech synthesis and voice conversion technologies, Audio Deepfake has become a serious threat to the Automatic Speaker Verification (ASV) system. Numerous countermeasures are proposed to detect this type of attack. In this paper, we report our efforts to combine the self-supervised WavLM model and Multi-Fusion Attentive classifier for audio deepfake detection. Our method exploits the WavLM model to extract features that are more conducive to spoofing detection for the first time. Then, we propose a novel Multi-Fusion Attentive (MFA) classifier based on the Attentive Statistics Pooling (ASP) layer. The MFA captures the complementary information of audio features at both time and layer levels. Experiments demonstrate that our methods achieve state-of-the-art results on the ASVspoof 2021 DF set and provide competitive results on the ASVspoof 2019 and 2021 LA set.

Target speaker extraction (TSE) aims to extract the target speaker’s voice from the input mixture. Previous studies have concentrated on high-overlapping scenarios. However, real-world applications usually meet more complex scenarios like variable speaker overlapping and target speaker absence. In this paper, we introduces a framework to perform continuous TSE (C-TSE), comprising a target speaker voice activation detection (TSVAD) and a TSE model. This framework significantly improves TSE performance on similar speakers and enhances personalization, which is lacking in traditional diarization methods. In detail, unlike conventional TSVAD deployed to refine the diarization results, the proposed Attention-target speaker voice activation detection (A-TSVAD) to directly generate timestamps of the target speaker. We also explore some different integration methods of A-TSVAD and TSE by comparing the cascaded and parallel methods. The framework’s effectiveness is assessed using a range of metrics, including diarization and enhancement metrics. Our experiments demonstrate that A-TSVAD outperforms conventional methods in reducing diarization errors, when integrating A-TSVAD and TSE in a sequential cascaded manner further enhances extraction accuracy. Audio demos are available on our demo page1.

Audio and visual signals usually coexist in realistic scenes, and human brains can learn this multi-modal perception easily. So, it is crucial for the computer to learn how human brains work for solving multi-modal tasks. The Audio-visual event localization (AVEL) task involves two sub-tasks: find video segments that contain Audio-visual events, and determine the category of the events. However, the AVEL task remains challenging due to the severe background noise. Additionally, processing information from both modalities simultaneously is also a tough issue. The current approaches have two main problems. One is that the network tends to be influenced by noise and predicts unreasonable events for consecutive segments within the same video clip. The other is that the model will oscillate between the local and global targets due to the multi-objective learning. To address these problems, we propose a decoupling multi-modal fusion network, which not only suppresses the complex noise but also learns the local and global information exclusively. The proposal consists of two sub-networks: the Which-event sub-network for predicting the event category and the Is-event sub-network for determining the time boundary for the event. We evaluate our method on the standard AVE Dataset in both fully and weakly supervised settings, and the results verify the effectiveness of our method.

Computer-Assisted Pronunciation Training (CAPT) plays an important role in language learning. Conventional ASR-based CAPT methods require expensive annotation of the ground truth pronunciation for the supervised training. Mean-while, certain undefined non-native phonemes cannot be correctly classified into standard phonemes, making the annotation process challenging and subjective. On the other hand, ASR-based CAPT methods only give the learner text-based feedback about the mispronunciation, but cannot teach the learner how to pronounce the sentence correctly. To solve these limitations, we propose to use the acoustic unit (AU) as the intermediary feature for both mispronunciation detection and correction. The proposed method uses the masked AU sequence and the target phonemes to detect the error AU and then corrects it. This method can give the learner speech-based self-imitating feedback, making our CAPT powerful for education.

Universal style transfer methods typically leverage rich representations from deep Convolutional Neural Network (CNN) models (e.g., VGG-19) pre-trained on large collections of images. Despite the effectiveness, its application is heavily constrained by the large model size to handle ultra-resolution images given limited memory. In this work, we present a new knowledge distillation method (named Collaborative Distillation) for encoder-decoder based neural style transfer to reduce the convolutional filters. The main idea is underpinned by a finding that the encoder-decoder pairs construct an exclusive collaborative relationship, which is regarded as a new kind of knowledge for style transfer models. Moreover, to overcome the feature size mismatch when applying collaborative distillation, a linear embedding loss is introduced to drive the student network to learn a linear embedding of the teacher’s features. Extensive experiments show the effectiveness of our method when applied to different universal style transfer approaches (WCT and AdaIN), even if the model size is reduced by 15.5 times. Especially, on WCT with the compressed models, we achieve ultra-resolution (over 40 megapixels) universal style transfer on a 12GB GPU for the first time. Further experiments on optimization-based stylization scheme show the generality of our algorithm on different stylization paradigms. Our code and trained models are available at https://github.com/mingsun-tse/collaborative-distillation.

Portrait matting is of vital importance for many applications such as portrait editing, background replacement, ecommerce demonstration, and augmented reality. The portrait matt can be accessed by predicting the α value of the original picture. Previous deep matting methods usually adopt a segmentation network to tackle portrait matting tasks. However, these traditional methods will introduce unpleasant blemishes in the matting results sometimes. The authors find that the key factor behind this phenomenon is how they model the matting problem. On the one hand, α value predicting can be modelled as a regression task. On the other hand, it can be viewed as a classification task of predicting background or foreground. To solve this problem, they explore different methods to model the nature of the α matting problem and propose a novel quantisation‐based adaption. Their method comes up with an α quantisation loss to achieve multi‐threshold filtering. Furthermore, they apply an α merging block to improve conventional regression methods. With their method, the gradient loss is reduced by 7.53% relatively, with mean square error and sum of absolute difference decreased by 14.7% relatively, leading to a more visually pleasant α matt in several segmentation backbones.

Parameter pruning is a promising approach for CNN compression and acceleration by eliminating redundant model parameters with tolerable performance degrade. Despite its effectiveness, existing regularization-based parameter pruning methods usually drive weights towards zero with large and constant regularization factors, which neglects the fragility of the expressiveness of CNNs, and thus calls for a more gentle regularization scheme so that the networks can adapt during pruning. To achieve this, we propose a new and novel regularization-based pruning method, named IncReg, to incrementally assign different regularization factors to different weights based on their relative importance. Empirical analysis on CIFAR-10 dataset verifies the merits of IncReg. Further extensive experiments with popular CNNs on CIFAR-10 and ImageNet datasets show that IncReg achieves comparable to even better results compared with state-of-the-arts. Our source codes and trained models are available here: https://github.com/mingsun-tse/caffe_increg.

This paper reports the newly designed programming course developed and taught at Zhejiang University for a freshman class majoring in information science and electronic engineering, and disseminated to many other universities, to address the enrollment crisis as reported by ACM Curriculum Committee Review Task Force. The course teaches the basic information science and electronic engineering concepts and gives the students practical programming experience through robotics programming. The initial curriculum and the ongoing improvement of the course are presented. The course started with its experiment environment using ASU Visual IoT/Robotics Programming Environment (VIPLE). The environment supports a variety of platforms, including open architecture-based robots, such as Intel robots, PCDuino robots, vendor-specific robots, such as Lego EV3, as well as simulated robots, such as Web simulated robots in HTML5 and Unity-based 3D robots.

Multiple scattering effects are commonly ignored in the detection and estimation of scatterers in signal processing research, because the energy of the first-order scattering is much larger than that of higher-order components. Although multiple scattering can significantly increase the estimation precision of point scatterers, it does not always lead to an improvement. Identifying conditions under which multiple scattering is beneficial or detrimental to estimation in a general setup is still an open problem. In this paper, we consider the effects of multiple scattering on the localization of two point scatterers. By comparing the Fisher information matrix on location parameters when multiple scattering exists and does not exist, we show analytically that information on ranges can benefit estimating directions of arrival via multiple scattering when the two scatterers are in far-field and well resolved.