This paper proposes a new approach for detecting tone errors in continuous Mandarin speech. In the training phase, tone variations are modeled with context-depended MSD-HMM which considers six contextual factors instead of two in traditional triphone HMM. In the evaluation phase, the goodness of tone pronunciation is measured by Kullback-Leibler divergence (KLD) between the expected tone model and the most representative tone model. When the KLD between the two models is larger than a threshold, the tone is detected as a pronunciation error. In the ROC curve, we get the equal error rate at 2.6%.

This paper presents the methods to improve the performance of mispronunciation detection at syllable level for Mandarin from two aspects: proposing scaled log-posterior probability (SLPP) and weighted phone SLPP to get the better measure of pronunciation quality; introducing speaker normalization of speaker adaptive training (SAT) and speaker adaptation of selective maximum likelihood linear regression (SMLLR) to get a better statistical model. Experiments based on a database, consisting of 8000 syllables pronounced by 40 speakers with varied pronunciation proficiency, confirm the promising effectiveness of these strategies by reducing FAR from 41.1% to 31.4% at 90% FRR and 36.0% to 16.3%at 95%FRR.

In this paper, we propose to convert the prosody boundary detection task into a syllable position labeling task. In order to detect both prosodic word and prosodic phrase boundaries, 6 types of syllable positions are defined. For each position, context-dependent position models are trained from manually labeled data. These models are used to label syllable positions in unseen speech. Word and phrase boundaries are then easily derived from syllable position labels. The proposed approach is tested with a large scale single speaker database. The precision and recall for word boundary are 96.1 % and 90.1%, respectively, and for phrase boundary are 83.7 % and 80.5%, respectively. Results of a listening test shows that only 28% of word boundaries and 50 % of phrase of boundaries detected automatically are critical error, implying only about 2.2 % and 10 % errors for word and phrase boundaries, respectively. The results are rather good, especially when it is considered that only acoustic features are used in this work. Index Terms: Boundary detection, context-dependent position model, prosodic word, phrase

To identify the language origin of a named entity, morphological information associated with its letter spelling, such as letter N-grams, is commonly employed. However, with this information only, named entities with similar spellings but from different language origins are difficult to differentiate. In this paper, a measure of "popularity," in terms of frequency or page count of the named entity in language-specific Web search, is proposed for identifying its language origin. Morphological information, including letter or letter-chunk N-grams, is used to enhance the performance of language identification in conjunction with Web-based page counts. Six languages, including English, German, French, Portuguese, Chinese, and Japanese (Chinese and Japanese named entities are shown in their corresponding phonetic alphabets, i.e., Pinyin and Romaji), are tested. Experiments show that when classifying four Latin languages, including English, German, French, and Portuguese, which are written in Latin alphabets, features from different information sources yield substantial performance improvements in the classification accuracy over a letter 4-gram-based baseline system. The accuracy increases from 75.0% to 86.3%, or a 45.2% relative error reduction.

Automatic accent annotation is important in both speech synthesis and speech recognition. Existing statistical learning algorithms rely heavily on a sufficiently large set of labeled training samples that are expensive and time consuming to collect. For unlabeled data, unsupervised learning can be initiated with a small set of manually labeled data. This paper shows that the accuracy of automatic accent annotation can be improved by augmenting a small amount of manually labeled data with a large pool of unlabeled data. We introduce an agreement-learning algorithm for this propose. Experimental results show that it is possible to reduce human-labeling effort significantly while reducing up to 50% errors.

In this paper, we investigate the automatic pronunciation evaluation method for native Mandarin. Multi-space distribution (MSD) hidden Markov model (HMM) is adopted to train the gold standard model. Machine scores derived from the generalized segment posterior probability on both syllables and phone level are proposed and investigated to measure the goodness of pronunciation (GOP). They are evaluated on the database collected internally and shown better performance than other well-known methods. In addition, detailed analyses of human scoring such as inter/intra-rater on utterance/speaker level are also given.

In this paper, we study tonal variations by training contextdependent tone models from a large speech corpus. Each model represents a tone-in-context and can output a stylized f0 pattern for it. With these tone models, it becomes tangible to investigate f0-variations with plenty of factors. Six contextual factors are investigated in this work. We find that impact of a factor varies across tones as well as the three states of a tone. Normally, onset pitch of a tone is determined interactively by syllable position and left tone, while, the offset pitch is mainly determined by syllable position. For a neutral tone, its pitch level is mainly depended on the left tone and syllable position affects its offset pitch. Both current vowel and right tone influence the pitch level, yet their impacts are weaker than syllable position and left tone, except for T3, the low tone. Index Terms: tonal variation, context-dependent, tone model 1.

Corpus-driven speech synthesis is hampered by the occurrence of occasional glitches which ruin the impression of the whole utterance. We propose an iterative unit selection integrated with an unnatural prosody detection model to identify any unnatural prosody. The system searches an optimal path in the lattice, verifies its naturalness by the unnatural prosody model and replaces the bad section with a better candidate, until it passes the verification test. In light of hypothesis testing, we show this trial-and-error approach takes effective advantage of abundant candidate samples in the database. Also, in contrast to conventional prosody prediction, an unnatural prosody detection model still leaves enough room for the prosody variations. Unnaturalness confidence measures are studied. The combined model can reduce the objective distortion by 16.3%. Perceptual experiments also confirm the proposed approach improves the synthetic speech quality appreciably. Index Terms: speech synthesis, unit selection, confidence measure, unnatural prosody detection, iterative synthesis

Identifying the language origin of a name in English is important for generating its correct pronunciation. In this paper, N-grams of syllable-based letter clusters are proposed for the task. The performance of the N-gram model of a set of frequently used letter clusters (correspond to syllables) is compared to that of letter N-gram model in a four-language task: English, German, French, and Portuguese. On average, the letter cluster N-gram, which has 26% error rate, is slightly better than the letter N-gram, which has 27.2% error rate. Furthermore, it is found that the error distributions from the two N-grams have fairly large differences. Therefore, AdaBoost is used to combine the results from N-grams of different units. The error rate is reduced to 22.5% or a relative 17.5% error reduction is achieved after the combination

This paper proposes a hierarchical framework, which consists of three layers of classifiers, for automatic stress detection in English speech utterances. The top two layers are a linguistic classifier, which assigns stressed labels to all content words and unstressed labels to all functions words, and an acoustic classifier, which assigns stressed and unstressed labels with HMM based models and using only acoustic features such as MFCC, energy and f0. When there is no manual stressed label available, only the top two layers are activated. The best performance we achieved is 92.9%. The third layer in the framework is an AdaBoost classifier that can improve the accuracy by using more features and manual labels. The best result we obtained is 94.1%, which is approaching to the self-agreement ratio (97.4%) of the same annotator, or the upper bound of the performance.

This paper proposes a new approach for measuring the target cost in unit selection, where the difference between the target and candidate units is estimated by the Kullback-Leibler divergence (KLD) between the context-dependent hidden Markov models (HMM). In order to model the left/right phonetic context, biphone models are generated by merging regular tri-phone HMMs sharing the same left/right phonetic context. To characterize prosodic contexts, various sets of prosody-sensitive monophone HMMs are trained. KLDs between these context models are calculated as the replacement cost between the contexts. Perceptual experiments show that the resulting synthesized speech sounds slightly better than those with the manually-tuned costs. An important advantage is that the proposed method can be conveniently applied to new corpora or languages without the need of collecting perceptual data

In this paper, we explore how to construct stylistic TTS databases from audio books, in which a storyteller performs multiple roles. The goal is to identify and build a set of speech corpora, each of which not only portrays a representative voice style performed by the speaker, but also has sufficient sentences to synthesize natural speech using unit selection approach. We solve the problem in two procedures: first, by representing each role with Gaussian Mixture Models (GMM), all speech data are partitioned into a number of voice style clusters with a criterion that maximizes the likelihood of all utterances with respect to roles’ speaker models; then, pruning in terms of both acoustic and prosodic measures is followed to purify the clusters. The resulting 4 voice styles are subjectively interpreted as Neutral, Young, Elder and Adult, respectively. Perceptual experiments show that the proposed approach can synthesize speech with the recognizable voice styles with an average 72.5% identification rate, and the synthesized speech sounds better than those synthesized with utterances from a single role.

An RF IC optimization methodology based on an elitist distributed particle swarm optimization algorithm is presented. By including a Pareto ranking mechanism and elitism in the algorithm, design alternatives and tradeoff information are provided with high efficiency. Post-optimization Monte-Carlo simulations are performed to assess first-order yield performance and aid in the selection of the final design. The approach is validated through the synthesis of a 5.2GHz direct-conversion front-end in 180nm CMOS.

A hierarchical divide-and-conquer multi-optimization methodology for phase-locked loop synthesis is presented. By optimizing each building block in the PLL separately with various optimization techniques, high optimization efficiency and good circuit performance are achieved. The methodology is validated with the synthesis of a 1GHz third-order PLL in 240nm SiGe BiCMOS.

In unit selection based speech synthesizer, defining a good unit set is crucial to the speech quality. In this paper, a method of customizing the TTS base unit set with a specific speech corpus is proposed. Multi-phoneme units are boosted from the initial phoneme-sized unit. A new multi-phoneme unit is added to the inventory based upon its own frequency count and the affected frequency count of other units. As a result, a large base unit set, which contains many multi-phoneme units, is formed when the speech corpus is large. While, for a small speech corpus, only a few bi-phoneme or tri-phoneme are found. Such a scalable base unit set makes it possible to achieve better smoothness in concatenation while maintain the naturalness of prosody. Evaluation results show that, after replacing the phone-sized base unit set with the customized set, the search speed is improved by 5 times and 59% preference score is obtained.

Accurate phonetic transcription is critical to high quality concatenation based text-to-speech synthesis. In this paper, we propose to use generalized syllable posterior probability (GSPP) as a statistical confidence measure to verify errors in phonetic transcriptions, such as reading errors, inadequate alternatives of pronunciations in the lexicon, letter-to-sound errors in transcribing out-of-vocabulary words, idiosyncratic pronunciations, etc. in a TTS speech database. GSPP is computed based upon a syllable graph generated by a recognition decoder. Testing on two data sets, the proposed GSPP is shown to be effective in locating phonetic transcription errors. Equal error rates (EERs) of 8.2% and 8.4%, are obtained on two testing sets, respectively. It is also found that the GSPP verification performance is fairly stable over a wide range around the optimal value of acoustic model exponential weight used in computing GSPP.

Consistent phoneme segmentation is essential in building high quality Text-to-Speech (TTS) voice fonts. In this paper we propose to adapt an existing well-trained Context Dependent Boundary Model (CDBM) for refining segment boundaries to a new speaker with limited, manually segmented data. Three adaptation approaches: MLLR, MAP, and a combination of the two, are studied. The combined one, MLLR+MAP, delivers the best boundary refinement performance. In comparison with other boundary segmentation methods, the adapted CDBM yields better results, especially with a limited amount of adaptation data. Given 400 manually segmented boundary tokens in about 20 sentences as a development set, the segmentation precision can reach 90% of human labeled boundaries within a tolerance of 20 ms.

Modeling units play a very important role in state-of-art speech recognition systems. The design and selection of them will directly impact the performance of the final speech recognition engine. As a tonal language, Mandarin's modeling units are more special for the tonal processing. In this paper, after fully investigating several dominant modeling strategies, we propose a new phone set design strategy for Mandarin, called segmental tonal modeling. Instead of modeling tone types directly, we realize them implicitly and jointly by two segments, which both carry tonal information. Both HTK and SAPI based experiments confirmed that such a method is very efficient. In addition to improving the accuracy by 9-23%, it greatly reduces the decoding time by 30-45%. Given the similar decoding speed, the new phone set configuration can reduce the error rate by relatively 35%.

Different feature extraction and tone modeling schemes are investigated on both speaker-dependent and speaker-independent continuous speech databases. Tone recognition features can be classified as detailed features which use the entire F0 curve, and outlined features which capture the main structure of the F0 curve. Tone models of different size, ranging from very simple one-tone-one-model tone models to complex phoneme-dependent tone models, have different abilities to characterize tone. Our experiments show two conclusions. First, the detailed information of the F0 curve is not necessary for tone recognition. The outlined features can, not only reduce the number of parameters, but also improve the accuracy of tone recognition. The proposed subsection average F0 and /spl Delta/F0 are shown to be effective outlined features. The second conclusion is that the one-tone-one-model scheme is not sufficient. Building phoneme-dependent tone models can highly improve the recognition accuracy, especially for speaker-independent data. Thus we suggest using fractionized models, trained with the outlined features, for tone recognition.

This paper proposed a post-refining method with fine contextual-dependent GMM for the auto-segmentation task. A GMM trained with a super feature vector extracted from multiple evenly spaced frames near the boundary is suggested to describe the waveform evolution across a boundary. CART is used to cluster acoustically similar GMM, so that the GMM for each leaf node is reliably trained by the limited manually labeled boundaries. An accuracy of 90% is thus achieved when only 250 manually labeled sentences are provided to train the refining models.

This paper describes a bilingual text-to-speech (TTS) system, Microsoft Mulan, which switches between Mandarin and English smoothly and which maintains the sentence level intonation even for mixed-lingual texts. Mulan is constructed on the basis of the Soft Prediction Only prosodic strategy and the Prosodic-Constraint Orient unit-selection strategy. The unit-selection module of Mulan is shared across languages. It is insensitive to language identity, even though the syllable is used as the smallest unit in Mandarin, and the phoneme in English. Mulan has a unique module, the language-dispatching module, which dispatches texts to the language-specific front-ends and merges the outputs of the two front-ends together. The mixed texts are "uttered" out with the same voice. According to our informal listening test, the speech synthesized with Mulan sounds quite natural. Sample waves can be heard at: http://research.microsoft.com/-echang/proiects/tts/mulan.htm.

This paper puts forward a domain adaptation problem that has not been studied well. For corpus-driven TTS systems, domain adaptation is realized by adding a small amount of domain-specific speech that will provide the maximum increase in average length of units that are used for synthesizing speech in that domain. An approach for generating optimized script for adaptation is proposed, the core of which is a dynamic programming based algorithm that segments domain-specific corpus into minimum number of segments that appear in the unit inventory. Increase in MOS after adaptation can be estimated from the generated script without recording speech from it. The results show that the amount of MOS increase depends not only on the size of the training set and the size of the script for adaptation, but also on the broadness of the domain. Narrower domains have larger increase in MOS.

This paper proposes a method for optimizing the cost function for unit selection in the corpus-based TTS system by maximizing the correlation between the concatenative cost and the MOS. To do this, a subjective evaluation should be done first. The key point is to log the contextual information of all units appearing in the synthetic utterances evaluated. With this log file, concatenative cost can be recalculated with a cost function in any new definition. Then, the correlation between cost and MOS can serve as a measure for the validity of any change in the cost function, and the cost function is optimized perpetually without any new MOS evaluation. In this paper, the correlation coefficient between cost and MOS improves from –0.822 to –0.897 after optimization. 1.

This paper proposes a channel equalization algorithm for a large speech database with application in concatenative TTS systems. The convolutional channel distortion is equalized by comparing the power spectral densities (PSDs) of utterances of different recording sessions. Autoregressive linear filters are designed on a corpus level and are used offline to filter the corresponding sentences to compensate for the relative distortions caused by the channel effects. Two experiments are carried out to evaluate the benefit of the channel equalization approach. First, this method is used to reduce the distance of their PSDs between two recording sessions to verify the effectiveness of the method. Secondly, it is applied practically in the TTS system. The whole TTS speech database is processed to reduce the PSDs variance over all sessions. Moreover, a subjective listening test is carried out to obtain human evaluation of the new TTS system. Almost all listeners prefer the synthetic speech generated by the new TTS system. Furthermore, an analysis of variance (ANOVA) on this subjective listening test demonstrates that the channel equalization process has significant effect on increasing the perceived voice-quality consistency of the TTS system.

This paper proposes a two-module text to speech system (TTS) structure, which bypasses the prosody model that predicts numerical prosodic parameters for synthetic speech. Instead, many instances of each basic unit from a large speech corpus are classified into categories by a classification and regression tree (CART), in which the expectation of the weighted sum of square regression error of prosodic features is used as splitting criterion. Better prosody is achieved by keeping slender diversity in prosodic features of instances belonging to the same class. A multi-tier non-uniform unit selection method is presented. It makes the best decision on unit selection by minimizing the concatenated cost of a whole utterance. Since the largest available and suitable units are selected for concatenating, distortion caused by mismatches at concatenated points is minimized. Very natural and fluent speech is synthesized, according to informal listening test.

This paper stresses the importance of converting a string of lexical words to that of prosodic words in text-to-speech (TTS) systems by presenting the surface differences and perceptual differences between them. A statistical rule based method and a classification and regression tree (CART) based method are proposed as solutions. Though ComplicatedSet based CART method performs the best, the achievement is obtained at the cost of heavy computation workloads needed by a parser. Statistical rule based method results in higher recall but lower precision, comparing to SimpleSet CART method. It is very difficult to tell which is better, since we don't know which affects naturalness more, precision or recall. Both of them require only lexicon word segmentation and part of speech (POS) tagging in the preprocessing stage, and are easily realized in TTS systems. Results of the preference test discloses that significant improvements on naturalness are perceived when lexical word strings are converted into prosodic word strings by our approach.

This paper studies factors that have influences on durations of syllables in Mandarin. Six factors are investigated by comparing means of duration in different categories in the preparatory study. The vector space is reduced from 299250 to 1000 by removing factors that do not cause significant changes in means and merging levels in each factor that have similar means. In order to remove influences from various initials and finals, durations of syllables are divided by the mean duration of the corresponding base syllables. The sum of squared residuals analysis is performed on the normalized duration. The results reveal that boundary index has the largest influence on syllable durations. Tone identity comes next. Though, position in word is not a very important single factor, it influences duration together with boundary index.