We present a dictionary-learning algorithm for reducing the sparse coding residual of an exemplar-based method for native-to-nonnative voice conversion (VC). The proposed algorithm iteratively updates the source and target speaker dictionaries to reduce both the residual and voice conversion error, thereby increasing synthesis quality. We evaluate the method on speech from the ARCTIC and L2-ARCTIC corpora and compare it to a baseline exemplar-based VC algorithm. The proposed algorithm significantly improves synthesis quality to more than double that of the baseline system while using two orders of magnitude fewer atoms. Additionally, the proposed algorithm significantly reduces both the VC error and the residual magnitude. We discuss the implications of the algorithm for broad exemplar-based VC systems.

Voice conversion (VC) techniques can be used to synthesize utterances from second language learners to appear as if they have a native accent, providing learners with an ideal target to imitate in pronunciation training. In prior work, we presented a low-resource technique called SABR (Sparse, Anchor-Based Representation of Speech), which uses acoustic anchorsone per English phonemeto represent an utterance as a sparse, linear combination of nonnegative weights. SABR produces intelligible speech, but its compact size limits the acoustic quality of the synthesis, in large part due to the significant residual left out by the compact model. In this article, we propose SABR+Res, which uses a linear combination of frequency warp transforms to convert the source residual spectrum to be closer to that of the target speaker and use it in synthesis. We evaluate the proposed method on speakers from the ARCTIC and L2-ARCTIC databases and compare them to state-of-the-art exemplar and frequency-warping VC methods. We find that SABR+Res had the lowest objective VC error for native-to-nonnative conversion and was preferred in subjective tests. Additionally, when compared to the baseline systems, SABR+Res had a much higher synthesis quality on native-to-nonnative speakers, performing similarly to native-to-native speaker pairs. We discuss the implications for the residual warping system and applying the residual transform to other exemplar-based systems.

Sparse-coding techniques for voice conversion assume that an utterance can be decomposed into a sparse code that only carries linguistic contents, and a dictionary of atoms that captures the speakers' characteristics. However, conventional dictionary-construction and sparse-coding algorithms rarely meet this assumption. The result is that the sparse code is no longer speaker-independent, which leads to lower voice-conversion performance. In this paper, we propose a Cluster-Structured Sparse Representation (CSSR) that improves the speaker independence of the representations. CSSR consists of two complementary components: a Cluster-Structured Dictionary Learning module that groups atoms in the dictionary into clusters, and a Cluster-Selective Objective Function that encourages each speech frame to be represented by atoms from a small number of clusters. We conducted four experiments on the CMU ARCTIC corpus to evaluate the proposed method. In a first ablation study, results show that each of the two CSSR components enhances speaker independence, and that combining both components leads to further improvements. In a second experiment, we find that CSSR uses increasingly larger dictionaries more efficiently than phoneme-based representations by allowing finer-grained decompositions of speech sounds. In a third experiment, results from objective and subjective measurements show that CSSR outperforms prior voice-conversion methods, improving the acoustic quality of the synthesized speech while retaining the target speaker's voice identity. Finally, we show that the CSSR captures latent (i.e., phonetic) information in the speech signal.

In previous work we presented a Sparse, Anchor-Based Representation of speech (SABR) that uses phonemic “anchors” to represent an utterance with a set of sparse non-negative weights. SABR is speaker-independent: combining weights from a source speaker with anchors from a target speaker can be used for voice conversion. Here, we present an extension of the original SABR that significantly improves voice conversion synthesis. Namely, we take the residual signal from the SABR decomposition of the source speaker's utterance, and warp it to the target speaker's space using a weighted warping function learned from pairs of source-target anchors. Using subjective and objective evaluations, we examine the performance of adding the warped residual (SABR+Res) to the original synthesis (SABR). Specifically, listeners rated SABR+Res with an average mean opinion score (MOS) of 3.6, a significant improvement compared to 2.2 MOS for SABR alone (p <; 0.01) and 2.5 MOS for a baseline GMM method (p <; 0.01). In an XAB speaker identity test, listeners correctly identified the identity of SABR+Res (81 %) and SABR (84%) as frequently as a GMM method (82%) (p = 0.70, P = 0.35). These results indicate that adding the warped residual can dramatically improve synthesis while retaining the desirable independent qualities of SABR models.

We describe a method for adapting a physical vocal tract model's anatomical and gestural parameters using acoustic information to match a target speaker. Physical vocal tract models are hard to adjust to match a speaker, as doing so requires information which is difficult to capture, such as X-Ray or MRI information. We propose an analysis-by-synthesis approach to adjust the parameters of the VocalTractLab (VTL) physical vocal tract model, optimizing on an acoustic distance objective function. We compare our method with one which does not adjust anatomy parameters, just gestural parameters, and find that the proposed method results in a net improvement. We also test our method's ability to recreate a synthetic speaker for which the ground truth parameters are known, and find that the method can reproduce the speaker if parameters pertaining to teeth and lips are fixed.

We present SABR (Sparse, Anchor-Based Representation), an analysis technique to decompose the speech signal into speaker-dependent and speaker-independent components. Given a collection of utterances for a particular speaker, SABR uses the centroid for each phoneme as an acoustic “anchor, ” then applies Lasso regularization to represent each speech frame as a sparse non-negative combination of the anchors. We illustrate the performance of the method on a speaker-independent phoneme recognition task and a voice conversion task. Using a linear classifier, SABR weights achieve significantly higher phoneme recognition rates than Mel frequency Cepstral coefficients. SABR weights can also be used directly to perform accent conversion without the need to train a speaker-to-speaker regression model.