This paper proposes language-independent prosody-enhanced speech representations to improve the naturalness of speech synthesis for the target languages that lack prosodic labels. To build text-to-speech (TTS) systems for low-resource languages, recent studies have employed the representations extracted from self-supervised learning (SSL) speech models, such as wav2vec 2.0, as intermediate representations in TTS models. However, they have generally focused only on the linguistic and phonetic information in SSL representations, disregarding the prosodic information. This paper investigates the prosodic information contained in the multilingual wav2vec 2.0 model through layer-wise probing tests utilizing acoustic prosodic features and prosodic labels. Furthermore, we propose a language-independent prosody enhancement approach to improve the prosodic properties of SSL models. The proposed method introduces a prosodic label prediction loss to fine-tune wav2vec 2.0 model with multilingual prosody-annotated corpora. From the fine-tuned wav 2 vec 2.0 model, the language-independent prosody-enhanced speech representations are extracted and serve as intermediate representations of our acoustic model in the downstream TTS task. The experimental results on six target languages demonstrate that our proposed prosody-enhanced speech representations outperform the original wav2vec 2.0 representations without enhancement.

This paper presents a multilingual speech synthesis approach that leverages learned phonetic representations to eliminate the need for pronunciation dictionaries in target languages. The learned phonetic representations consist of unsupervised phonetic representations (UPR) and supervised phonetic representations (SPR). To extract UPRs, a pre-trained wav2vec 2.0 model is utilized, while a language-independent automatic speech recognition (LI-ASR) model with a connectionist temporal classification (CTC) loss is employed to derive segment-level SPRs from the speech data of target languages. An acoustic model using UPRs and SPRs as intermediate representations is then designed, comprising a UPR predictor, an SPR predictor, and a representation-to-mel-spectrogram (RTM) converter. The two predictors generate UPRs and SPRs from texts, respectively. The RTM converter first combines UPRs with SPRs using a Transformer-based encoder, and then feeds the merged representations into a decoder to produce mel-spectrograms. Considering the difficulty of collecting large training corpora for all languages in multilingual speech synthesis, the parameters of both the two predictors and the RTM converter can be pre-trained on non-target languages to further improve model performance. Experimental results on six target languages demonstrate that our method outperformed the approaches directly predicting mel-spectrograms from character or phoneme sequences, and pre-training the acoustic model using a multilingual corpus further improved the performance of synthetic speech.

This paper proposes a multilingual speech synthesis method which combines unsupervised phonetic representations (UPR) and supervised phonetic representations (SPR) to avoid reliance on the pronunciation dictionaries of target languages.In this method, a pretrained wav2vec 2.0 model is adopted to extract UPRs and a language-independent automatic speech recognition (LI-ASR) model is built with a connectionist temporal classification (CTC) loss to extract segment-level SPRs from the audio data of target languages.Then, an acoustic model is designed, which first predicts UPRs and SPRs from texts separately and then combines the predicted UPRs and SPRs to generate mel-spectrograms.The results of our experiments on six languages show that the proposed method outperformed the methods that directly predicted mel-spectrograms from character or phoneme sequences and the ablated models that utilized only UPRs or SPRs.