A hierarchical pitch model (HPM) was recently proposed to HMM-based speech synthesis. In HPM, pitch trajectory is modeled as an additive combination of hierarchical layers (including state, phone, syllable, etc), and a minimum generation error (MGE) criterion is used to re-estimate model parameters. In this paper, we extend the MGE criterion to a tree-based model clustering process to simultaneously cluster the context-dependent models at all layers, and construct a full MGE training process for HPM training. Experiments were conducted to investigate the effects of HPM with different training criteria and different hierarchical layer combinations. Experimental results show that the full MGE training can significantly improve HPM's ability to predict F0 trajectory in TTS over the ML-based approach on test data. The new HPM also outperforms the conventional state-level HMM in F0 prediction.

In this paper, we propose a minimum generation error (MGE) training method to refine the audio-visual HMM to improve visual speech trajectory synthesis. Compared with the traditional maximum likelihood (ML) estimation, the proposed MGE training explicitly optimizes the quality of generated visual speech trajectory, where the audio-visual HMM modeling is jointly refined by using a heuristic method to find the optimal state alignment and a probabilistic descent algorithm to optimize the model parameters under the MGE criterion. In objective evaluation, compared with the ML-based method, the proposed MGE-based method achieves consistent improvement in the mean square error reduction, correlation increase, and recovery of global variance. It also improves the naturalness and audio-visual consistency perceptually in the subjective test.

In conventional speech synthesis, large amounts of phonetically balanced speech data recorded in highly controlled recording studio environments are typically required to build a voice. Although using such data is a straightforward solution for high quality synthesis, the number of voices available will always be limited, because recording costs are high. On the other hand, our recent experiments with HMM-based speech synthesis systems have demonstrated that speaker-adaptive HMM-based speech synthesis (which uses an “average voice model” plus model adaptation) is robust to non-ideal speech data that are recorded under various conditions and with varying microphones, that are not perfectly clean, and/or that lack phonetic balance. This enables us to consider building high-quality voices on “non-TTS” corpora such as ASR corpora. Since ASR corpora generally include a large number of speakers, this leads to the possibility of producing an enormous number of voices automatically. In this paper, we demonstrate the thousands of voices for HMM-based speech synthesis that we have made from several popular ASR corpora such as the Wall Street Journal (WSJ0, WSJ1, and WSJCAM0), Resource Management, Globalphone, and SPEECON databases. We also present the results of associated analysis based on perceptual evaluation, and discuss remaining issues.

This paper proposes a state duration modeling method using full covariance matrix for HMM-based speech synthesis. In this method, a full covariance matrix instead of the conventional diagonal covariance matrix is adopted in the multi-dimensional Gaussian distribution to model the state duration of each context-dependent phoneme. At synthesis stage, the state durations are predicted using the clustered context-dependent distributions with full covariance matrices. Experimental results show that the synthesized speech using full-covariance state duration models is more natural than the conventional method when we change the speaking rate of synthesized speech.

This paper improves a minimum generation error (MGE) based HMM training technique for HMM-based speech synthesis by directly using the original spectrum instead of line spectral pairs (LSPs) as reference spectrum for log spectral distortion (LSD) measure. Two types of original reference spectra for LSD calculation are investigated, including the spectrum extracted from speech waveform by STRAIGHT, and the short-time FFT spectrum calculated from speech waveforms. Since only the harmonics of the FFT spectrum are coincident with the underlying spectral envelope, the LSD between generated LSPs and original FFT spectrum is calculated by sampling at the harmonic frequencies, and a weighting function is designed to simulate the sampling strategy on LSPs. From the experimental results, the MGE-LSD training using the FFT spectrum as reference spectrum achieved the best performance.

A phone mapping-based method had been introduced for cross-lingual speaker adaptation in HMM-based speech syn-thesis. In this paper, we continue to propose a state mapping based method for cross-lingual speaker adaptation, where the state mapping between voice models in source and target lan-guages is established under minimum Kullback-Leibler diver-gence (KLD) criterion. We introduce two approaches to use the established mapping information for cross-lingual speaker adaptation, including data mapping and transform mapping ap-proaches. From the experimental results, the state mapping based method outperformed the phone mapping based method. In addition, the data mapping approach achieved better speaker similarity, and the transform mapping approach achieved better speech quality after cross-lingual speaker adaptation. Index Terms: Speech synthesis, HMM, speaker adaptation, minimum generation error, linear regression

A minimum generation error (MGE) criterion had been proposed for model training in HMM-based speech synthesis. In this paper, we apply the MGE criterion to model adaptation for HMM-based speech synthesis, and introduce an MGE linear regression (MGELR) based model adaptation algorithm, where the regression matrices used to transform source models are optimized so as to minimize the generation errors of adaptation data. In addition, we incorporate the recent improvements of MGE criterion into MGELR-based model adaptation, including state alignment under MGE criterion and using a log spectral distortion (LSD) instead of Euclidean distance for spectral distortion measure. From the experimental results, the adaptation performance was improved after incorporating these two techniques, and the formal listening tests showed that the quality and speaker similarity of synthesized speech after MGELRbased adaptation were significantly improved over the original MLLR-based adaptation. Index Terms: Speech synthesis, HMM, speaker adaptation, minimum generation error, linear regression

Due to the inconsistency between the maximum likelihood (ML) based training and the synthesis application in HMM-based speech synthesis, a minimum generation error (MGE) criterion had been proposed for HMM training. This paper continues to apply the MGE criterion to model adaptation for HMM-based speech synthesis. We propose a MGE linear regression (MGELR) based model adaptation algorithm, where the regression matrices used to transform source models to target models are optimized to minimize the generation errors for the input speech data uttered by the target speaker. The proposed MGELR approach was compared with the maximum likelihood linear regression (MLLR) based model adaptation. Experimental results indicate that the generation errors were reduced after the MGELR-based model adaptation. And from the subjective listening test, the discrimination and the quality of the synthesized speech using MGELR were better than the results using MLLR.

Two techniques, including minimum generation error (MGE) criterion for HMM training, and the parameter generation algorithm considering global variance (GV), had been proposed to improve the quality of HMM-based speech synthesis. In this paper, we incorporate the GV technique into MGE criterion, where an additional generation error component considering global/local variance (GV/LV) is introduced for generation error definition, and the model parameters are optimized to minimize the new generation error function. From the experimental results, the quality of synthesized speech was improved after MGE-GV/LV training, which is similar to the effectiveness of considering GV in parameter generation, however, without introducing any extra computational cost in synthesis process.

A minimum generation error (MGE) criterion had been proposed to solve the issues related to maximum likelihood (ML) based HMM training in HMM-based speech synthesis. In this paper, we improve the MGE criterion by imposing a log spectral distortion (LSD) instead of the Euclidean distance to define the generation error between the original and generated line spectral pair (LSP) coefficients. Moreover, we investigate the effect of different sampling strategies to calculate the integration of the LSD function. From the experimental results, using the LSDs calculated by sampling at LSPs achieved the best performance, and the quality of synthesized speech after the MGE-LSD training was improved over the original MGE training.

In order to solve the issues related to the maximum likelihood (ML) based HMM training for HMM-based speech synthesis, a minimum generation error (MGE) criterion had been proposed. This paper continues to apply the MGE criterion to model adaptation for HMM-based speech synthesis. We introduce a MGE linear regression (MGELR) based model adaptation algorithm, where the transforms from source HMMs to target HMMs are optimized to minimize the generation errors for the adaptation data of the target speaker. The regression matrices for both mean vector and covariance matrix of Gaussian distribution are re-estimated. The proposed MGELR approach was compared with the maximum likelihood linear regression (MLLR) based model adaptation. Experimental results indicate that the generation errors were reduced after the MGELR-based model adaptation. And from the subjective listening test, the speaker similarity and the quality of the synthesized speech using MGELR were better than the results using MLLR.

In maximum-likelihood (ML) based HMM synthesis, the generated trajectory of a sentence in the training set is in general does not reproduce the trajectory of the original one. To overcome this shortcoming, a minimum generation error (MGE) criterion has been previously proposed. In this paper, a complete MGE-based HMM training is introduced, where the MGE criterion is applied to the entire training process, including context-dependent HMM training, context-dependent HMM clustering and clustered HMM training. In this procedure, the HMMs are trained to minimize the generation error of training data, which is in line with the HMM-based synthesis. From the experiments, the quality of synthesized speech is improved after applying the MGE criterion to the whole training process.

In modeling online handwritten characters, imaginary strokes have been conveniently generated by connecting adjacent real strokes together to form a continuous trajectory. However, this approach causes confusions among characters with similar but actually different trajectories. In this paper, we propose to use multi-space probability distribution (MSD) to model imaginary strokes jointly with real strokes. With the proposed MSD, real and imaginary strokes become observations from different probability spaces and they are modeled stochastically. Also, the flexibility in MSD to assign different feature dimensions to each individual space enables us to ignore certain features that can cause singularity problem in modeling. Experimental results obtained in handwritten Chinese character recognition indicate MSD provides 1.3%-2.8% character recognition accuracy improvement across different recognition systems where MSD significantly improves discrimination among confusable characters with similar trajectories.

In HMM-based speech synthesis, there are two issues critical related to the MLE-based HMM training: the inconsistency between training and synthesis, and the lack of mutual constraints between static and dynamic features. In this paper, we propose minimum generation error (MGE) based HMM training method to solve these two issues. In this method, an appropriate generation error is defined, and the HMM parameters are optimized by using the generalized probabilistic descent (GPD) algorithm, with the aims to minimize the generation errors. From the experimental results, the generation errors were reduced after the MGE-based HMM training, and the quality of synthetic speech is improved.

Abstract To improve the performance of the HMM-based voice conversion system in which the LSP coefficient is introduced as the spectral representation, a model clustering technique to tie HMMs into classes for the model adaptation, considering the phonetic and linguistic contextual factors of HMMs, is adopted in this paper. Besides, due to the relationship between the LSP coefficients of adjacent orders, an appropriate format of the regression matrix is suggested according to the small amount of the adaptation training data. Subjective and objective tests prove that the source HMMs can be adapted more accurately using the proposed method, meanwhile the synthetic speech generated from the adapted model has better discrimination and speech quality. Index Terms : model adaptation, regression matrix clustering, and regression matrix format 1. Introduction With the development of the corpus-based speech synthesis technique, the intelligibility and naturalness of the synthetic speech has been improved a lot. However, it is still a difficult problem for the corpus-based TTS system to synthesize speech of various speakers and speaking styles with a limited database. So the voice conversion technique which can convert one speaker’s voice to another speaker’s voice provides a positive approach to achieve the goal of synthesizing speech of multi-speakers. The HMM-based voice conversion system is built on the basis of the HMM-based speech synthesis. In the HMM-based speech synthesis system, spectrum, pitch and duration are modeled simultaneously in a unified framework of HMMs [1][2][3]. In addition, voice characteristics of the synthetic speech can be converted from one speaker to another by applying a model adaptation algorithm, such as the MLLR (maximum likelihood linear regression) algorithm [4][5], with a small amount of speech uttered by the target speaker. We have realized a HMM-based speech synthesis system in which the LSP (line spectral pair) coefficients and the STRAIGHT (Speech Transformation and Representation using Adaptive Interpolation of weighted spectral contour) analysis-synthesis algorithm are introduced [6][7]. Then, by realizing the MLLR algorithm, we provide our synthesis system with the ability of synthesizing voice of various speakers. However, there still exist two main problems in the HMM-based voice conversion system. Firstly, the data-driven clustering method described in the MLLR algorithm ignores many contextual factors between HMMs, therefore some unrelated HMMs are forced into one class which will affect the accuracy of the model adaptation. Secondly, the system performance including the voice characteristics and voice quality of the synthetic speech decreases greatly when the adaptation training data is very limited. In order to solve these problems, a clustering method, considering the phonetic and linguistic connections between HMMs using the context decision tree, which has been applied similarly in both the HMM-based speech recognition and the HMM-based speech synthesis areas [8][9], is described in this paper. Moreover, an appropriate regression matrix format is suggested when very few training data is available, as the LSP coefficients of only several adjacent orders have strong correlations. In the following part of this paper, an overview of our HMM-based voice conversion system is presented in section 2. Section 3 describes the details of the proposed context clustering decision tree and the appropriate regression matrix for the model adaptation. Section 4 presents the results of experiments including subjective and objective evaluations while section 5 provides a final conclusion.

In the conventional HMM-based segmentation method, the HMM training is based on MLE criteria, which links the segmentation task to the problem of distribution estimation. The HMM are built to identify the phonetic segments, not to detect the boundary. This kind of inconsistency between training and application limited the performance of segmentation. In this paper, we adopt the discriminative training method and introduce a new criterion, named minimum segmentation error (MSGE), for HMM training. In this method, a loss function directly related to the segmentation error is defined. By minimizing the overall empirical loss with the generalized probabilistic descent (GPD) algorithm, the segmentation error is also minimized. From the results on both Chinese and Japanese data, the accuracy of segmentation is improved. Moreover, this method is robust even when we do not have enough knowledge on HMM modeling, e.g. the number of states is not optimized.

In corpus-based speech synthesis, the quality of the synthetic speech critically depends on the speech corpus. Since the high vowel in Japanese might be devoiced in the real speech, we should detect and transcribe them automatically in the corpus construction. In this paper, we apply the HMM-based method, and adopt two kinds of likelihood differences as voicing measures for different focuses. To improve the detection performance, the discriminative training is applied to voiced/ devoiced HMM training. Moreover, some features that can discriminate the voiced/devoiced units, including duration, energy and autocorrelation, are incorporated together with the likelihood differences in several methods. The experiments show different results for each high vowel, i.e. the devoicing is vowel dependent. For the vowel /i/, the discriminative training can improve the detection performance to a certain degree. And by cumulating the voicing features and the likelihood differences with optimized weights, the detection accuracy is improved. But for the vowel /u/, there is very limited improvement, even with the voicing features.