Infrared and visible image fusion aims to describe the same scene from different aspects by combining complementary information of multi-modality images. The existing Generative adversarial networks (GAN) based infrared and visible image fusion methods cannot perceive the most discriminative regions, and hence fail to highlight the typical parts existing in infrared and visible images. To this end, we integrate multi-scale attention mechanism into both generator and discriminator of GAN to fuse infrared and visible images (AttentionFGAN). The multi-scale attention mechanism aims to not only capture comprehensive spatial information to help generator focus on the foreground target information of infrared image and background detail information of visible image, but also constrain the discriminators focus more on the attention regions rather than the whole input image. The generator of AttentionFGAN consists of two multi-scale attention networks and an image fusion network. Two multi-scale attention networks capture the attention maps of infrared and visible images respectively, so that the fusion network can reconstruct the fused image by paying more attention to the typical regions of source images. Besides, two discriminators are adopted to force the fused result keep more intensity and texture information from infrared and visible image respectively. Moreover, to keep more information of attention region from source images, an attention loss function is designed. Finally, the ablation experiments illustrate the effectiveness of the key parts of our method, and extensive qualitative and quantitative experiments on three public datasets demonstrate the advantages and effectiveness of AttentionFGAN compared with the other state-of-the-art methods.

This paper proposes a unit-selection and waveform concatenation speech synthesis system based on synthetic speech naturalness evaluation. A Support Vector Machine (SVM) and Log Likelihood Ratio (LLR) based synthetic speech naturalness evaluation system was introduced in our previous work. In this paper, the evaluation system is improved in three aspects. Finally, a unit-selection and concatenation waveform speech synthesis system is built on the base of the synthetic speech naturalness evaluation system. Optimum unit sequence is chosen through the re-scoring for the N-best path. Subjective listening tests show the proposed synthetic speech evaluation based speech synthesis system significantly outperforms the traditional unit-selection speech synthesis system.

In this paper, we present a novel approach to relax the constraint of stereo-data which is needed in a series of algorithms for noise-robust speech recognition. As a demonstration in SPLICE algorithm, we generate the pseudo-clean features to replace the ideal clean features from one of the stereo channels, by using HMM-based speech synthesis. Experimental results on aurora2 database show that the performance of our approach is comparable with that of SPLICE. Further improvements are achieved by concatenating a bias adaptation algorithm to handle unknown environments. Relative word error rate reductions of 66% and 24% are achieved over the baseline systems in the clean-training and multi-training conditions, respectively.

This paper proposes a method to detect the errors in synthetic speech of a unit selection speech synthesis system automatically using log likelihood ratio and support vector machine (SVM). For SVM training, a set of synthetic speech are firstly generated by a given speech synthesis system and their synthetic errors are labeled by manually annotating the segments that sound unnatural. Then, two context-dependent acoustic models are trained using the natural and unnatural segments of labeled synthetic speech respectively. The log likelihood ratio of acoustic features between these two models is adopted to train the SVM classifier for error detection. Experimental results show the proposed method is effective in detecting the errors of pitch contour within a word for a Mandarin speech synthesis system. The proposed SVM method using log likelihood ratio between context-dependent acoustic models outperforms the SVM classifier trained on acoustic features directly.

This paper proposes a state duration modeling method using full covariance matrix for HMM-based speech synthesis. In this method, a full covariance matrix instead of the conventional diagonal covariance matrix is adopted in the multi-dimensional Gaussian distribution to model the state duration of each context-dependent phoneme. At synthesis stage, the state durations are predicted using the clustered context-dependent distributions with full covariance matrices. Experimental results show that the synthesized speech using full-covariance state duration models is more natural than the conventional method when we change the speaking rate of synthesized speech.

Gaussian mixture models (GMM) have become one of the standard acoustic approaches for language identification. Furthermore, the GMM-SVM is proven to work well by introducing the discriminative method into the GMM-based acoustic systems. In these systems, the intersession variability within language has become an important adverse factor that degrades the system performance. To tackle this problem, we propose a subspace analysis method, termed as Intra-language Difference Subspace Estimatio (IDSE), under the GMM-SVM framework. In IDSE method, the difference vector is modeled with three components: Extra-language difference, Intra-language difference and noise difference. Then the Intra-language and noise difference are effectively estimated and eliminated from the difference vector. The experiments on NIST 07 evaluation tasks show effectiveness of the proposed method.

This paper presents an investigation into ways of integrating articulatory features into hidden Markov model (HMM)-based parametric speech synthesis. In broad terms, this may be achieved by estimating the joint distribution of acoustic and articulatory features during training. This may in turn be used in conjunction with a maximum-likelihood criterion to produce acoustic synthesis parameters for generating speech. Within this broad approach, we explore several variations that are possible in the construction of an HMM-based synthesis system which allow articulatory features to influence acoustic modeling: model clustering, state synchrony and cross-stream feature dependency. Performance is evaluated using the RMS error of generated acoustic parameters as well as formal listening tests. Our results show that the accuracy of acoustic parameter prediction and the naturalness of synthesized speech can be improved when shared clustering and asynchronous-state model structures are adopted for combined acoustic and articulatory features. Most significantly, however, our experiments demonstrate that modeling the dependency between these two feature streams can make speech synthesis systems more flexible. The characteristics of synthetic speech can be easily controlled by modifying generated articulatory features as part of the process of producing acoustic synthesis parameters.

Heteroscedastic discriminant analysis (HDA) with two-dimensional (2D) constraints is proposed in this paper. HDA suffers from the small sample size problem and instability when lack of training data or feature dimension is high, even when the number of dimension is in a suitable range. Two-dimensional HDA is first proposed, then we show that 2D methods are actually a kind of structure-constrained 1D methods, and lastly, HDA with 2D constraints is proposed. Experiments on TIMIT and WSJ0 show that the proposed method outperforms other methods.

In this paper, we propose a new feature normalization approach for robust speech recognition. It is found that the shape of speech feature distributions is changed in noisy environments compared with that in the clean condition. So cepstral shape normalization (CSN) which normalizes the shape of feature distributions is performed by exploiting an exponential factor. This method has been proven effective in noisy environments, especially under low SNRs. Experimental results show that the proposed method yields relative word error rate reductions of 38% and 25% on aurora2 and aurora3 databases, respectively, in comparing with those of the conventional mean and variance normalization (MVN). It is also shown CSN consistently outperforms other traditional methods, such as histogram equalization (HEQ) and higher order cepstral moment normalization (HOCMN).

This paper presents a minimum unit selection error (MUSE) training method for HMM-based unit selection speech synthesis system, which selects the optimal phone-sized unit sequence from the speech database by maximizing the combined likelihood of a group of trained HMMs. Under MUSE criterion, the weights and distribution parameters of these HMMs are estimated to minimize the number of different units between the selected phone sequences and the natural phone sequences for the training sentences. The optimization is realized by discriminative training using generalized probabilistic descent (GPD) algorithm. Results of our experiment show that this proposed method is able to improve the performance of the baseline system where model weights are set manually and distribution parameters are trained under maximum likelihood criterion.

Due to the inconsistency between the maximum likelihood (ML) based training and the synthesis application in HMM-based speech synthesis, a minimum generation error (MGE) criterion had been proposed for HMM training. This paper continues to apply the MGE criterion to model adaptation for HMM-based speech synthesis. We propose a MGE linear regression (MGELR) based model adaptation algorithm, where the regression matrices used to transform source models to target models are optimized to minimize the generation errors for the input speech data uttered by the target speaker. The proposed MGELR approach was compared with the maximum likelihood linear regression (MLLR) based model adaptation. Experimental results indicate that the generation errors were reduced after the MGELR-based model adaptation. And from the subjective listening test, the discrimination and the quality of the synthesized speech using MGELR were better than the results using MLLR.

Two techniques, including minimum generation error (MGE) criterion for HMM training, and the parameter generation algorithm considering global variance (GV), had been proposed to improve the quality of HMM-based speech synthesis. In this paper, we incorporate the GV technique into MGE criterion, where an additional generation error component considering global/local variance (GV/LV) is introduced for generation error definition, and the model parameters are optimized to minimize the new generation error function. From the experimental results, the quality of synthesized speech was improved after MGE-GV/LV training, which is similar to the effectiveness of considering GV in parameter generation, however, without introducing any extra computational cost in synthesis process.

This paper presents a novel discriminative training criterion, minimum word classification error (MWCE). By localizing conventional string-level MCE loss function to word-level, a more direct measure of empirical word classification error is approximated and minimized. Because the word-level criterion better matches performance evaluation criteria such as WER, an improved word recognition performance can be achieved. We evaluated and compared MWCE criterion in a unified DT framework, with other commonly-used criteria including MCE, MMI, MWE, and MPE. Experiments on TIMIT and WS JO evaluation tasks suggest that word-level MWCE criterion can achieve consistently better results than string-level MCE. MWCE even outperforms other substring-level criteria on the above two tasks, including MWE and MPE.

This paper presents the methods to improve the performance of mispronunciation detection at syllable level for Mandarin from two aspects: proposing scaled log-posterior probability (SLPP) and weighted phone SLPP to get the better measure of pronunciation quality; introducing speaker normalization of speaker adaptive training (SAT) and speaker adaptation of selective maximum likelihood linear regression (SMLLR) to get a better statistical model. Experiments based on a database, consisting of 8000 syllables pronounced by 40 speakers with varied pronunciation proficiency, confirm the promising effectiveness of these strategies by reducing FAR from 41.1% to 31.4% at 90% FRR and 36.0% to 16.3%at 95%FRR.

We continue our previous work on soft margin estimation (SME) to large vocabulary continuous speech recognition (LVCSR) in two new aspects. The first is to formulate SME with different unit separation. SME methods focusing on string-, word-, and phone-level separation are defined. The second is to compare SME with all the popular conventional discriminative training (DT) methods, including maximum mutual information estimation (MMIE), minimum classification error (MCE), and minimum word/phone error (MWE/MPE). Tested on the 5k-word Wall Street Journal task, all the SME methods achieves a relative word error rate (WER) reduction from 17% to 25% over our baseline. Among them, phone-level SME obtains the best performance. Its performance is slightly better than that of MPE, and much better than those of other conventional DT methods. With the comprehensive comparison with conventional DT methods, SME demonstrates its success on LVCSR tasks.

Abstract � This paper presents a method to control the characteristics of synthetic speech flexibly by integrating articulatory features into a Hidden Markov Model (HMM)-based parametric speech synthesis system. In contrast to model adaptation and interpolation approaches for speaking style control, this method is driven by phonetic knowledge, and target speech samples are not required. The joint distribution of parallel acoustic and articulatory features considering cross-stream feature dependency is estimated. At synthesis time, acoustic and articulatory features are generated simultaneously based on the maximum-likelihood criterion. The synthetic speech can be controlled flexibly by modifying the generated articulatory features according to arbitrary phonetic rules in the parameter generation process. Our experiments show that the proposed method is effective in both changing the overall character of synthesized speech and in controlling the quality of a specific vowel. Index Terms: speech synthesis, hidden Markov model, articulatory features, phonetic knowledge

In order to solve the issues related to the maximum likelihood (ML) based HMM training for HMM-based speech synthesis, a minimum generation error (MGE) criterion had been proposed. This paper continues to apply the MGE criterion to model adaptation for HMM-based speech synthesis. We introduce a MGE linear regression (MGELR) based model adaptation algorithm, where the transforms from source HMMs to target HMMs are optimized to minimize the generation errors for the adaptation data of the target speaker. The regression matrices for both mean vector and covariance matrix of Gaussian distribution are re-estimated. The proposed MGELR approach was compared with the maximum likelihood linear regression (MLLR) based model adaptation. Experimental results indicate that the generation errors were reduced after the MGELR-based model adaptation. And from the subjective listening test, the speaker similarity and the quality of the synthesized speech using MGELR were better than the results using MLLR.

In this paper, we propose a novel optimization algorithm called constrained line search (CLS) for discriminative training (DT) of Gaussian mixture continuous density hidden Markov model (CDHMM) in speech recognition. The CLS method is formulated under a general framework for optimizing any discriminative objective functions including maximum mutual information (MMI), minimum classification error (MCE), minimum phone error (MPE)/minimum word error (MWE), etc. In this method, discriminative training of HMM is first cast as a constrained optimization problem, where Kullback-Leibler divergence (KLD) between models is explicitly imposed as a constraint during optimization. Based upon the idea of line search, we show that a simple formula of HMM parameters can be found by constraining the KLD between HMM of two successive iterations in an quadratic form. The proposed CLS method can be applied to optimize all model parameters in Gaussian mixture CDHMMs, including means, covariances, and mixture weights. We have investigated the proposed CLS approach on several benchmark speech recognition databases, including TIDIGITS, Resource Management (RM), and Switchboard. Experimental results show that the new CLS optimization method consistently outperforms the conventional EBW method in both recognition performance and convergence behavior.

We extend our previous work on soft margin estimation (SME) to large vocabulary continuous speech recognition in two aspects. The first is to use the extended Baum-Welch method to replace the conventional generalized probabilistic descent algorithm for optimization. The second is to compare SME with minimum classification error (MCE) training with the same implementation details in order to show that it is indeed the margin component in the objective function with margin-based utterance and frame selection that contributes to the success of SME. Tested on the 5 k-word Wall Street Journal task, all the SME methods work better than MCE. The best SME approach achieves a relative word error rate reduction of about 19% over our best baseline performance. This enhancement can only be demonstrated because of our use of margin-based objective function and the extended Baum-Welch parameter optimization method.

Recently, we proposed a novel optimization algorithm called constrained line search (CLS) to train Gaussian mean vectors of HMMs in the MMI sense. In this paper, we extend and re-formulate it in a more general framework. The new CLS can optimize any discriminative objective functions including MMI, MCE, MPE/MWE etc. Also, closed-form solutions to update all Gaussian mixture parameters, including means, covariances and mixture weights, are obtained. We investigate the new CLS on several benchmark speech recognition databases, including TIDIGITS, Switchboard mini-train and Switchboard full h5train00 sets. Experimental results show that the new CLS optimization method outperforms the conventional EBW method in both performance and convergence behavior.

In this paper, we investigate locality preserving projections (LPP) in two-dimensional sense. Recently, LPP was proposed for dimensionality reduction, which can detect the intrinsic manifold structure of data and preserve the local information. When image data are concerned, they are often vectorized for LPP. However, the dimension of image data is usually very high, LPP can't be implemented due to singularity of matrix. We propose two methods for image dimensionality reduction: two-dimensional LPP (2DLPP) and bilateral two-dimensional LPP (B2DLPP), which are based directly on 2D image matrices rather than 1D vectors as LPP does. Experiments are conducted on the ORL face database, which shows higher recognition performance of the proposed methods.

We propose to use minimum divergence, where acoustic similarity between HMMs is characterized by Kullback-Leibler divergence, for discriminative training. The MD objective function is defined as a posterior weighted divergence measured over the whole training set. Different from our earlier work, where KLD-based acoustic similarity is pre-computed for all initial models and stays invariant in the optimization procedure, here we propose to jointly optimize the whole variable MD by adjusting HMM parameters since MD is a function of the adjusted HMM parameters. An EBW optimization method is derived to minimize the whole MD objective function. The new MD formulation is evaluated on the TIDIGITS and Switchboard databases. Experimental results show that the new MD yields relative word error rate reductions of 62.1% on TIDIGITS and 8.8% on Switchboard databases when compared with the best ML-trained systems. It is also shown the new MD consistently outperforms other discriminative training criteria, such as MPE.

This paper presents a hidden Markov model (HMM) based unit selection method using hierarchical units under statistical criterion. In our previous work we tried to use frame sized speech segments and maximum likelihood criterion to improve the performance of traditional concatenative synthesis system using phone sized units and cost function criterion. In this paper, hierarchical units which consist of phone level units and frame level units are adopted to achieve better balance between the coverage rate of candidate unit and the number of concatenation points during synthesis. Besides, Kullback-Leibler divergence (KLD) between candidate and target phoneme HMMs is introduced as a part of the final criterion for unit selection. The listening result proves that these two approaches can improve the performance of synthetic speech effectively.

In this paper, we propose a novel constrained line search to optimize the MMEE objective function for training discriminative HMMs. In our method, the MMI estimation is cast as a constrained maximization problem, where Kullback-Leibler divergence between models before and after parameters adjustment is introduced as a constraint during optimization. Then, based on the idea of line search, we show that a simple, closed-form solution can be derived under some approximation assumptions. The proposed optimization method have been investigated in two speech recognition tasks: TIDIGITS and Switchboard (mini-train). Experimental results show that the new training method achieves significant word error rate reduction when comparing with our best MLE models, i.e., relatively 63.8% on TIDIGITS and 6.1% on the Switchboard mini-train set, respectively. Our results also show that the constrained line search method consistently outperforms the popular EBW method in both tasks.

Active learning and semi-supervised learning methods are frequently applied in multimedia annotation tasks in order to reduce human labeling effort. However, in most of these methods only single modality is applied. This paper presents an interactive video annotation framework, which is based on semi-supervised learning and active learning with multiple multimodalities. In the proposed framework, unlabeled samples are iteratively selected to be annotated manually according to certain strategy which has taken the potentials of different modalities into account, and then a graph-based semi-supervised learning algorithm is conducted on each modality. This process repeats for several rounds, and the results obtained from multiple modalities are then fused to generate final output. The proposed framework is computationally efficient, and the experimental results on TRECVID 2005 benchmark show that the proposed framework considerably outperforms previous approaches.

This paper introduces a tone pronunciation error detection algorithm for Mandarin CALL system. HMM is introduced to build tone model. FO after CDF-matching normalization is used as the feature of tone model. Tone error detection algorithm is based on the posterior probability calculated from HMM tone models. Comparing to the other normalization methods, CDF-matching can get better tone error detection performance with CC's increasing from 0.76 to 0.79, which is close to that between human evaluators (0.83).

In maximum-likelihood (ML) based HMM synthesis, the generated trajectory of a sentence in the training set is in general does not reproduce the trajectory of the original one. To overcome this shortcoming, a minimum generation error (MGE) criterion has been previously proposed. In this paper, a complete MGE-based HMM training is introduced, where the MGE criterion is applied to the entire training process, including context-dependent HMM training, context-dependent HMM clustering and clustered HMM training. In this procedure, the HMMs are trained to minimize the generation error of training data, which is in line with the HMM-based synthesis. From the experiments, the quality of synthesized speech is improved after applying the MGE criterion to the whole training process.

This paper presents a word graph based feature enhancement method for robust speech recognition in noise. The approach uses signal processing based speech enhancement as a starting point, and then performs Wiener filtering to remove residual noise. During the process, a decoded word graph is used to directly guide the feature enhancement with respect to the HMM for recognition, so that the enhanced feature can match the clean speech model better in the acoustic space. The proposed word graph based feature enhancement method was tested on the Aurora 2 database. Experimental results show that an improved recognition performance can be obtained comparing with conventional signal processing based and GMM based feature enhancement methods. With signal processing based weighted noise estimation and GMM based method, the relative error rate reductions are 35.44% and 42.58%, respectively. The proposed word graph based method improves the performance further, and a relative error rate reduction of 57.89% is obtained.

A new, state divergence-based algorithm is proposed in this paper to determine the number of Gaussian components of each state in continuous density HMM by maximizing the between-state divergence. The unscented transform based approximation of the Kullback-Leibler divergence is adopted to measure the between-state model divergence to direct the determination. Due to the advantage of being more discriminative, the proposed approach can lead to more compact HMM. Our experimental evaluation shows that compared with the conventional Bayesian Information Criterion based determination (which is better than the uniform determination), the presented method can reduce the total number of Gaussian components to about 63%, while it results in almost negligible degradation of the recognition performance.

Given a large set of video database, how to connect video segments with a certain set of semantic concepts with least manual labors is an elementary step for video indexing and searching. Due to the large gap between high-level semantics and low-level features, automatic video annotation with high accuracy is a challenging task. In this paper, we propose a novel automatic video annotation framework, which improves the annotation performance by learning from unlabeled samples and exploring temporal relationship in video sequences. To effectively learn from unlabeled data, a sample selection scheme based on combining a set of complementary predictors is proposed, which iteratively refines the performance of the initial predictors. A filtering-based method is applied to further improve the annotation accuracy as well, in which video temporal relationship is sufficiently exploited. Experiment results show that the proposed automatic video annotation method performs superior to general supervised learning methods and co-training

In HMM-based speech synthesis, there are two issues critical related to the MLE-based HMM training: the inconsistency between training and synthesis, and the lack of mutual constraints between static and dynamic features. In this paper, we propose minimum generation error (MGE) based HMM training method to solve these two issues. In this method, an appropriate generation error is defined, and the HMM parameters are optimized by using the generalized probabilistic descent (GPD) algorithm, with the aims to minimize the generation errors. From the experimental results, the generation errors were reduced after the MGE-based HMM training, and the quality of synthetic speech is improved.

Supervised and semi-supervised learning are frequently applied methods to annotate videos by mapping low-level features into semantic concepts. Due to the large semantic gap, the main constraint of these methods is that the information contained in a limited-size labeled dataset can hardly represent the distributions of the semantic concepts. In this paper, we propose a novel semi-automatic video annotation framework, active learning with semi-supervised ensembling, which tries to tackle the disadvantages of current video annotation solutions. Firstly the initial training set is constructed based on distribution analysis of the entire video dataset and then an active learning scheme is combined into a semi-supervised ensembling framework, which selects the samples to maximize the margin of the ensemble classifier based on both labeled and unlabeled data. Experimental results show that the proposed method performs superior to general semi-supervised learning algorithms and typical active learning algorithms in terms of annotation accuracy and stability

We propose to use Minimum Divergence(MD) as a new measure of errors in discriminative training. To focus on improving discrimination between any two given acoustic models, we refine the error definition in terms of Kullback-Leibler Divergence (KLD) between them. The new measure can be regarded as a modified version of Minimum Phone Error (MPE) but with a higher resolution than just a symbol matching based criterion. Experimental recognition results show the new MD based training yields relative word error rate reductions of 57.8% and 6.1% on TIDigits and Switchboard databases, respectively, in comparing with the ML trained baseline systems. The recognition performance of MD is also shown to be consistently better than that of MPE.

Abstract To improve the performance of the HMM-based voice conversion system in which the LSP coefficient is introduced as the spectral representation, a model clustering technique to tie HMMs into classes for the model adaptation, considering the phonetic and linguistic contextual factors of HMMs, is adopted in this paper. Besides, due to the relationship between the LSP coefficients of adjacent orders, an appropriate format of the regression matrix is suggested according to the small amount of the adaptation training data. Subjective and objective tests prove that the source HMMs can be adapted more accurately using the proposed method, meanwhile the synthetic speech generated from the adapted model has better discrimination and speech quality. Index Terms : model adaptation, regression matrix clustering, and regression matrix format 1. Introduction With the development of the corpus-based speech synthesis technique, the intelligibility and naturalness of the synthetic speech has been improved a lot. However, it is still a difficult problem for the corpus-based TTS system to synthesize speech of various speakers and speaking styles with a limited database. So the voice conversion technique which can convert one speaker’s voice to another speaker’s voice provides a positive approach to achieve the goal of synthesizing speech of multi-speakers. The HMM-based voice conversion system is built on the basis of the HMM-based speech synthesis. In the HMM-based speech synthesis system, spectrum, pitch and duration are modeled simultaneously in a unified framework of HMMs [1][2][3]. In addition, voice characteristics of the synthetic speech can be converted from one speaker to another by applying a model adaptation algorithm, such as the MLLR (maximum likelihood linear regression) algorithm [4][5], with a small amount of speech uttered by the target speaker. We have realized a HMM-based speech synthesis system in which the LSP (line spectral pair) coefficients and the STRAIGHT (Speech Transformation and Representation using Adaptive Interpolation of weighted spectral contour) analysis-synthesis algorithm are introduced [6][7]. Then, by realizing the MLLR algorithm, we provide our synthesis system with the ability of synthesizing voice of various speakers. However, there still exist two main problems in the HMM-based voice conversion system. Firstly, the data-driven clustering method described in the MLLR algorithm ignores many contextual factors between HMMs, therefore some unrelated HMMs are forced into one class which will affect the accuracy of the model adaptation. Secondly, the system performance including the voice characteristics and voice quality of the synthetic speech decreases greatly when the adaptation training data is very limited. In order to solve these problems, a clustering method, considering the phonetic and linguistic connections between HMMs using the context decision tree, which has been applied similarly in both the HMM-based speech recognition and the HMM-based speech synthesis areas [8][9], is described in this paper. Moreover, an appropriate regression matrix format is suggested when very few training data is available, as the LSP coefficients of only several adjacent orders have strong correlations. In the following part of this paper, an overview of our HMM-based voice conversion system is presented in section 2. Section 3 describes the details of the proposed context clustering decision tree and the appropriate regression matrix for the model adaptation. Section 4 presents the results of experiments including subjective and objective evaluations while section 5 provides a final conclusion.

In Chinese TTS (text-to-speech) systems, intonational phrase prediction has a great influence on the naturalness of synthesized speech. Different kinds of statistical models have been applied to this domain, and achieved good performance. We first build a maximum entropy model to yield the probability of each word boundary to be an intonational phrase break, and then a sliding window smoothing algorithm is proposed, in which the length distribution curve of the intonational phrase acts as the sliding window. The maximum entropy model and the distribution curve are trained from 19,000 sentences and tested on a test set of 1,000 sentences. Experimental results show that the sliding window smoothing algorithm makes an improvement of 5.3% in terms of F-score, 10.0% in terms of average score, and 55.6% in terms of unacceptable rate. From the results, we draw the conclusion that the length distribution information is of great usefulness for intonational phrase break prediction, and the sliding window smoothing method is quite effective in improving the performance significantly.

We propose an algorithm for optimal clustering and nonuniform allocation of Gaussian kernels in scalar (feature) dimension to compress complex, Gaussian mixture-based, continuous density HMMs into computationally efficient, small footprint models. The symmetric Kullback-Leibler divergence (KLD) is used as the universal distortion measure and it is minimized in both kernel clustering and allocation procedures. The algorithm was tested on the resource management (RM) database. The original context-dependent HMMs can be compressed to any resolution, measured by the total number of clustered scalar kernel components. Good trade-offs between the recognition performance and model complexities have been obtained; the HMM can be compressed to 15-20% of the original model size, which needs 1-5% of multiplication/division operations, and results in almost negligible recognition performance degradation.

In our approach, we first collect a competing token set for each physical HMM from training data. An off-line token collection procedure is used in this work to collect the competing-tokens from word lattices. Then we re-estimate HMM parameters discriminatively to minimize the total number of competing tokens counted in the phone level. The phone token counts are approximated by a sigmoid-based objective function. The GPD algorithm is used to adjust HMM parameters to minimize the objective function. In this work, a merging mechanism and a gradient normalization in the HMM tied-state level are proposed to improve the generalization power of our discriminative training method. The proposed method is evaluated on the resource management (RM) and the switchboard (a 24-hr mini-train set) tasks. Experimental results clearly show that our new discriminative training method achieves significant improvements over our best MLE models in both tasks, namely about 8% and 4.5% relative error rate reduction in RM and switchboard respectively, over the best MLE models.

In this paper, the minimum classification error (MCE) method is extended to optimize both linear discriminant analysis (LDA) transformation and the classification parameters for dimensionality reduction. Firstly, under the HMM-based continuous speech recognition (CSR) framework, we use the MCE criterion to optimize the conventional dimensionality reduction method, which uses LDA to transform the standard MFCC. Then, a new dimensionality reduction method is proposed. In the new method, the combination of discrete cosine transform (DCT) and LDA, as used in the conventional method, is replaced by a single LDA transformation, which is optimized according to MCE criterion along with the classification parameters. Experimental results on TiDigits show that even when the feature dimension is reduced to 14, the performance of this new method is as good as that of the MCE-trained system using 39 dimension MFCC. It also outperforms our MCE-optimized conventional dimensionality reduction method.

In October 2002, the advanced front-end (AFE) for distributed speech recognition (DSR) was standardized by ETSI. In order to use the AFE feature on low computational resource devices, we propose a novel approach to improve the computational efficiency. In our new algorithm, the structure of the two-stage mel-warped Wiener filtering algorithm, which is the main part of AFE, is modified. A Wiener filter is constructed and applied directly in the mel-warped filter-bank domain. The measures we take make many time-consuming operations in the original algorithm completely unnecessary, including the re-calculations of power spectrum and the time-domain convolution operations. Consequently, a large amount of computations are saved. Experiments show that the new approach can substantially reduce the computation load while preserving the excellent performance of the ETSI AFE.

In the conventional HMM-based segmentation method, the HMM training is based on MLE criteria, which links the segmentation task to the problem of distribution estimation. The HMM are built to identify the phonetic segments, not to detect the boundary. This kind of inconsistency between training and application limited the performance of segmentation. In this paper, we adopt the discriminative training method and introduce a new criterion, named minimum segmentation error (MSGE), for HMM training. In this method, a loss function directly related to the segmentation error is defined. By minimizing the overall empirical loss with the generalized probabilistic descent (GPD) algorithm, the segmentation error is also minimized. From the results on both Chinese and Japanese data, the accuracy of segmentation is improved. Moreover, this method is robust even when we do not have enough knowledge on HMM modeling, e.g. the number of states is not optimized.

Traditional video coding and streaming algorithms have not been fully utilized by using video content information for bit allocation and adaptation. This paper proposes a content-based video streaming method, based on a visual attention model to better utilize network bandwidth and achieve better subjective video quality. First, the visual attention model is exploited to segment the regions of interest (ROI) in video frames. Then, considering that the ROI is more sensitive to coding error than other regions, a region-weighted rate-distortion model is developed to allocate suitable bits for all ROI and non-ROI regions. The evaluation results indicate that more than 60% of the test video sequences encoded by the proposed method can obtain better subjective visual quality compared to the video encoded by classical methods under the same bandwidth, and about 20% of them can not be distinguished.

A maximum entropy based model for prosody phrase break prediction was proposed in this paper, and a comparison was conducted on large corpora between the new model and the decision tree based model which was the mainstream method for prosody phrase break prediction. The contribution of lexical information and influences of different cutoff values were also investigated. It was demonstrated that, utilizing the same information, maximum entropy based method made an improvement of 5.5% on F-Score over decision tree based method. Integrating lexical information, an improvement of 9.4% over decision tree was achieved. Using maximum entropy based method, to achieve the performance of traditional decision tree, 83% manual work could be saved.

This paper presents an alternative solution for speech database compression aiming at the embedded application of concatenative synthesis systems. The waveform of a speech segment is firstly decomposed into a prosodic pattern and a spectral pattern by STRAIGHT – a powerful speech analysissynthesis algorithm. Then all the prosodic and spectral patterns are clustered respectively to remove the redundant acoustic information within database. The clustering process is controllable and can export flexible compression ratio to meet the actual footprint requirement of various embedded devices. Besides, some labeling and contextual information are utilized to improve the performance of pattern clustering. Subjective listening test shows that our Mandarin synthesis system with corpus compressed by proposed method at about 2.7kbps perform corresponding to the same system compressed by G.723.1 at 5.3kps and the quality degradation is not serious as the compression ratio increases.

This paper presents a novel voice conversion system based on codebook mapping. A new phoneme-tied weighting strategy is proposed to reduce the smoothing effects in weighted sum of code books, while a new prosodic conversion method by decision tree is proposed to cope with the complex prosody of Chinese. STRAIGHT algorithm is used to decompose spectrum and excitation for separate modification. Listening tests prove the proposed methods can effectively convert speaker’s individuality while maintaining high speech quality with a small amount of training data.

In corpus-based speech synthesis, the quality of the synthetic speech critically depends on the speech corpus. Since the high vowel in Japanese might be devoiced in the real speech, we should detect and transcribe them automatically in the corpus construction. In this paper, we apply the HMM-based method, and adopt two kinds of likelihood differences as voicing measures for different focuses. To improve the detection performance, the discriminative training is applied to voiced/ devoiced HMM training. Moreover, some features that can discriminate the voiced/devoiced units, including duration, energy and autocorrelation, are incorporated together with the likelihood differences in several methods. The experiments show different results for each high vowel, i.e. the devoicing is vowel dependent. For the vowel /i/, the discriminative training can improve the detection performance to a certain degree. And by cumulating the voicing features and the likelihood differences with optimized weights, the detection accuracy is improved. But for the vowel /u/, there is very limited improvement, even with the voicing features.

Distributed speech synthesis (DSS) is a novel field of study and will be playing an active role in the domain of speech technology in the post-PC era. This paper focuses on the architecture design, implementation mechanism and optimization considerations of DSS systems. First, client-server based DSS architectures are investigated in server, client and network parts, respectively. Second, related system implementation issues are discussed correspondingly. Finally, system optimization suggestions in the aspects of data security, QoS control and load balancing between DSS server and client through dynamic arbitration are presented.

Sub-band speech recognition approaches have been proposed for robust speech recognition, where full-band power spectra are divided into several sub-bands and then likelihoods or cepstral vectors of the sub-bands are merged depending on their reliability. In conventional sub-band approaches, correlations across the sub-bands are not modeled and the merging weights can only be set experientially or estimated during training procedures, which may not match observed data. The methods further degrade performance for clean speech. We proposed a novel sub-band approach, where frequency sub-bands are multiplied with weighting factors and merged, which considers sub-band dependence and proves to be more robust than both full-band and conventional sub-band approaches. And further the weighting factors can be obtained by using the maximum-likelihood estimation approaches in order to minimize the mismatch between the trained models and the observed features. Finally we evaluated our methods on both the Aurora2 task and the Resource Management task and showed improvement of performance on the two tasks consistently. 1.

Because classical fast vector quantization(VQ) algorithms can’t be used in the LSF vector quantizers that use varying weighted Euclidean distance, a novel fast VQ search algorithm —CRVQCS (Constrained Range Vector Quantization based on Component Searching) is presented in this paper. The CRVQ-CS algorithm works well with the varying weighted Euclidean distance and yields the same result as full search VQ with reduced computational complexity does. Although the CRVQ-CS algorithm is proposed for VQ using varying weighted Euclidean distance measure, it is also suitable for VQ using simple Euclidean distance measure.

A weighted cepstral distance measure is proposed and tested in a speaker recognition system using a speaker-based vector quantization (VQ) approach. Based on the fine structure of the feature vector space, a statistically optimized distance measure is defined with weights equal to the partition-normalized inverse variance of cepstral coefficients. The weights can be adjusted individually for each partition and each component of the feature vector across all codebooks (speakers). Experiments on a 50-speaker database show that the suggested weighted cepstral distance measure works substantially better than the Euclidean cepstral distance or the inverse variance weighted cepstral distance. An accuracy of about 90% is achieved using a 16-level codebook in speaker verification.>