Personalized TTS is an exciting and highly desired application that allows users to train their TTS voice using only a few recordings. However, TTS training typically requires many hours of recording and a large model, making it unsuitable for deployment on mobile devices. To overcome this limitation, related works typically require fine-tuning a pre-trained TTS model to preserve its ability to generate high-quality audio samples while adapting to the target speaker’s voice. This process is commonly referred to as "voice cloning." Although related works have achieved significant success in changing the TTS model’s voice, they are still required to fine-tune from a large pre-trained model, resulting in a significant size for the voice-cloned model. In this paper, we propose applying trainable structured pruning to voice cloning. By training the structured pruning masks with voice-cloning data, we can produce a unique pruned model for each target speaker. Our experiments demonstrate that using learnable structured pruning, we can compress the model size to 7 times smaller while achieving comparable voice-cloning performance.

User-defined keyword spotting is a task to detect new spoken terms defined by users. This can be viewed as a few-shot learning problem since it is unreasonable for users to define their desired keywords by providing many examples. To solve this problem, previous works try to incorporate self-supervised learning models or apply meta-learning algorithms. But it is unclear whether self-supervised learning and meta-learning are complementary and which combination of the two types of approaches is most effective for few-shot keyword discovery. In this work, we systematically study these questions by utilizing various self-supervised learning models and combining them with a wide variety of meta-learning algorithms. Our result shows that HuBERT combined with Matching network achieves the best result and is robust to the changes of few-shot examples.

This paper proposes a content-based artist recommendation framework which learns relationships between users' preference and music contents through ordinal regression. In particular, an artist is characterized by the parameters of its corresponding acoustical model which is adapted from a universal background model. These artist-specific acoustic features together with their preference rankings are then used as input vectors for the proposed order preserving projection (OPP) algorithm which tries to find a suitable subspace such that the desired ranking order of the data after projection can be kept as much as possible. The proposed linear OPP can be kernelized to learn the nonlinear relationship between music contents and users' artist rank orders. Under the proposed framework of kernelized OPP (KOPP), we can derive the nonlinear relationship and, more importantly, efficiently fuse acoustic and symbolic features obtained from the artist recommended meta-data. Experimental results demonstrate that OPP attains comparable results with those obtained with a conventional ordinal regression method, Prank. Moreover, by exploring the nonlinear relationship among training examples and combining acoustic and symbolic features, KOPP outperforms previous approaches to artist recommendation.

Music can be viewed as a sequence of sound events. However, most of current approaches to genre classification either ignore temporal information or only capture local structures within the music under analysis. In this paper, we propose the use of a song tokenization method (which transforms the music into a sequence of units) in conjunction with a data mining technique for investigating the long-term structures (also known as sequential patterns) for music genre classification. Experimental results show that the introduction of sequential patterns can effectively outperform previous approach that considers local temporal features only for music genre classification.

Query-by-semantic-description (QBSD) is a natural way for searching/annotating music in a large database. To improve QBSD, we propose the use of anti-words for each annotation word based on the concept of supervised multiclass labeling (SML). More specifically, words that are highly associated with the opposite semantic meaning of a word constitute its anti-word set. By modeling both a word and its anti-word set, our annotation system can achieve 31.1% of equal mean per-word precision and recall, while the original SML model achieves 27.8%. Moreover, by constructing the models of the anti-word explicitly, the performance is also significantly improved for the retrieval system, especially when the query keyword is the antonym of an existing annotation word.