We present a Bayesian framework to learn prior and posterior distributions for latent variable models. Our goal is to deal with model regularization and achieve desirable prediction using heterogeneous speech data. A variational Bayesian expectation-maximization algorithm is developed to establish a latent variable model based on the exponential family distributions. This algorithm does not only estimate model parameters but also their hyperparameters which reflect the model uncertainties. The uncertainty is compensated to construct a variety of regularized models. We realize this full Bayesian framework for uncertainty decoding of speech signals. Compared to maximum likelihood method and Bayesian approach with heuristically-selected hyperparameters, the proposed method achieves higher speech recognition accuracy especially in case of sparse and noisy training data.

We present an evidence Bayesian framework, which can learn both the prior distributions and posterior distributions from data, for continuous-density hidden Markov models (CDHMM). The goal of this study is to build the regularized CDHMMs to improve model generalization, and achieve desirable recognition performance for unknown test speech. Under this framework, we develop an EM iterative procedure to estimate the marginal distribution or the evidence function for exponential family distributions. By adopting the variational Bayesian inference, we derive an empirical Bayesian solution to CDHMM parameters and their hyperparameters. Such a regularized CDHMM compensates the model uncertainty and the ill-posed conditions. Compared with maximum likelihood (ML) or other Bayesian approaches with heuristic hyperparameters, the proposed approach can utilize available data more effectively. The experiments on noisy speech recognition using Aurora2 show that the proposed Bayesian approach performs better than the baseline ML CDHMMs especially with mismatched test data or limited training data.

In this letter, we reformulate the discriminative training (DT) of continuous density hidden Markov models (CDHMMs) as an ellipsoid constrained quadratic programming (ECQP) problem, and we solve it by a line search algorithm. The ellipsoid constraint intrinsically arises from the step-size control in each iteration of DT optimization, which leads to an efficient solution without relaxing the objective function to be convex, as in a general quadratic programming problem. Moreover, the problem can be equivalently converted to a lower-dimensional one under some conditions, which helps further to simplify the solution. We show that under a Kullback–Leibler divergence (KLD) constraint, DT of CDHMM parameters such as Gaussian means and variances can be efficiently solved by the proposed algorithm, with only mild assumptions adopted. Experimental results on two tasks show that the ECQP approach considerably outperforms other popular algorithms in terms of both final recognition accuracy and convergence speed.

We study a specificpartialhypothesisfusionproblem in sequential data labeling. The problem arises in the multimodal applications where a decision is made by merging complete hypothesis from one input and partial hypothesis from the other. For example, in a pen-aided speech interface, appropriate pen input can provide partial but crucial information. We address the problem in a Bayesian framework, and reformulate the solution as a revised search in a representation. A dynamic programming algorithm is proposed to efficiently solve the partial hypothesis fusion via the graph. It is shown that the computational cost of the graph based partial hypothesis fusion is proportional to the size of the graph, which is highly feasible for a given compact graph. We apply the proposed algorithm to two real applications: an intelligent pen-based dictation error correction system and an automatic handwritten character completion with a speech ldquoshortcutrdquo. Experimental results show that the algorithm is effective in utilizing the partial information from one modality to enhance the bimodal interface performance.

In this paper, we propose to use discriminative training (DT) for improving letter-to-sound (LTS) conversion performance. LTS is a critical component in both ASR and TTS for predicting the correct pronunciation of a word not included in the lexicon. For TTS applications, predicting the proper pronunciation of an out-of-vocabulary person/place name, especially a name with foreign origin can be challenging. We utilize discriminative training, which has been successfully used in speech recognition, to sharpen the baseline N-grams of grapheme-phoneme pairs. We address the problem in a unified framework of discriminative training. Two criteria, maximum mutual information (MMI) and minimum phoneme error (MPE), are investigated. Experimental results show that DT yields a small (3.8-4.6% relative) but consistent error reduction across all databases tested. In addition, we observe that by pinpointing the local errors in a finer resolution, we can obtain a better discriminative model.

We address the problem of character auto-completion (CAC) for predicting Chinese characters with partial, cursive handwriting input. A prefix tree decoder based CAC algorithm is proposed. The approach is based upon HMM pen trajectory modeling and radical structure of Chinese characters. Without finishing the strokes, high quality character candidates can be efficiently predicted. As a result, significant improvement of the recognition throughput can be obtained. We combine further the handwriting CAC with speech recognition candidates in the posterior sense, and come up with a flexible, rapid bi- modal Chinese character input system. The system was tested on a large Chinese corpus and shown that: more than 90% of input attempts can be correctly finished with only 50% of the whole character trajectory written.

We study models that characterize pen trajectories of online handwritten characters in a fine manner. We propose radical based fine trajectory hidden Markov models (HMMs), which adopt radicals as basic units, and a multi-path HMM topology that emits observations with multi-space distributions (MSD) is built for each radical. Meanwhile, various stroke orders, writing styles and realness of sub-strokes are reasonably modeled. The radical based fine trajectory HMMs lead to handwriting recognition with effective prediction, and their generative nature can be utilized for a novel handwriting synthesis framework. Experimental show that along with the model precision increasing, about 50% recognition error can be reduced, and the fine models can generate decent character samples.

More mobile devices are starting to use automatic speech recognition for command or text input. However, correcting recognition errors in a small compact mobile device is usually inconvenient and it may take several finger operations on a small keypad to correct errors. In this paper, we propose a new multimodal input method and a novel confidence measure ― template constrained posterior (TCP) to simplify the correction process. The method works by interactively integrating a handwriting recognizer with a speech recognizer. Information obtained in pen-based error marking, like error location, error type, etc., is fed back to the speech recognizer, and speech recognition errors are automatically corrected using the TCP confidence measure. Experimental results on Aurora2, Wall Street Journal, Switchboard, and two Chinese databases show that compared with speech recognition baseline, the proposed method achieves relative error reduction of 64.9%, 43.9%, 26.1%, 39.0%, 31.4%, respectively, after the auto correction.

In this paper, we propose a novel optimization algorithm called constrained line search (CLS) for discriminative training (DT) of Gaussian mixture continuous density hidden Markov model (CDHMM) in speech recognition. The CLS method is formulated under a general framework for optimizing any discriminative objective functions including maximum mutual information (MMI), minimum classification error (MCE), minimum phone error (MPE)/minimum word error (MWE), etc. In this method, discriminative training of HMM is first cast as a constrained optimization problem, where Kullback-Leibler divergence (KLD) between models is explicitly imposed as a constraint during optimization. Based upon the idea of line search, we show that a simple formula of HMM parameters can be found by constraining the KLD between HMM of two successive iterations in an quadratic form. The proposed CLS method can be applied to optimize all model parameters in Gaussian mixture CDHMMs, including means, covariances, and mixture weights. We have investigated the proposed CLS approach on several benchmark speech recognition databases, including TIDIGITS, Resource Management (RM), and Switchboard. Experimental results show that the new CLS optimization method consistently outperforms the conventional EBW method in both recognition performance and convergence behavior.

Recently, we proposed a novel optimization algorithm called constrained line search (CLS) to train Gaussian mean vectors of HMMs in the MMI sense. In this paper, we extend and re-formulate it in a more general framework. The new CLS can optimize any discriminative objective functions including MMI, MCE, MPE/MWE etc. Also, closed-form solutions to update all Gaussian mixture parameters, including means, covariances and mixture weights, are obtained. We investigate the new CLS on several benchmark speech recognition databases, including TIDIGITS, Switchboard mini-train and Switchboard full h5train00 sets. Experimental results show that the new CLS optimization method outperforms the conventional EBW method in both performance and convergence behavior.

We propose a novel, divergence-based similarity measure for spoken document retrieval (SDR). We derive a dynamic programming algorithm that measures Kullback-Leibler divergence between two HMMs first. The measure is further generalized to a graph matching algorithm, which is efficient for SDR application. The proposed approach compares the underlying acoustic models of keywords and a target database to alleviate the impact of mismatched vocabulary and language model, e.g. different domains. Experimental results on the Wall Street Journal (WSJ) database show that the proposed approach achieves a comparable performance, compared with the word posterior based approach. It outperforms the latter when there is a mismatch in language model. The approach is promising for building an open-vocabulary, domain independent SDR application.

We propose to use minimum divergence, where acoustic similarity between HMMs is characterized by Kullback-Leibler divergence, for discriminative training. The MD objective function is defined as a posterior weighted divergence measured over the whole training set. Different from our earlier work, where KLD-based acoustic similarity is pre-computed for all initial models and stays invariant in the optimization procedure, here we propose to jointly optimize the whole variable MD by adjusting HMM parameters since MD is a function of the adjusted HMM parameters. An EBW optimization method is derived to minimize the whole MD objective function. The new MD formulation is evaluated on the TIDIGITS and Switchboard databases. Experimental results show that the new MD yields relative word error rate reductions of 62.1% on TIDIGITS and 8.8% on Switchboard databases when compared with the best ML-trained systems. It is also shown the new MD consistently outperforms other discriminative training criteria, such as MPE.

In this paper, we propose a novel constrained line search to optimize the MMEE objective function for training discriminative HMMs. In our method, the MMI estimation is cast as a constrained maximization problem, where Kullback-Leibler divergence between models before and after parameters adjustment is introduced as a constraint during optimization. Then, based on the idea of line search, we show that a simple, closed-form solution can be derived under some approximation assumptions. The proposed optimization method have been investigated in two speech recognition tasks: TIDIGITS and Switchboard (mini-train). Experimental results show that the new training method achieves significant word error rate reduction when comparing with our best MLE models, i.e., relatively 63.8% on TIDIGITS and 6.1% on the Switchboard mini-train set, respectively. Our results also show that the constrained line search method consistently outperforms the popular EBW method in both tasks.

In modeling online handwritten characters, imaginary strokes have been conveniently generated by connecting adjacent real strokes together to form a continuous trajectory. However, this approach causes confusions among characters with similar but actually different trajectories. In this paper, we propose to use multi-space probability distribution (MSD) to model imaginary strokes jointly with real strokes. With the proposed MSD, real and imaginary strokes become observations from different probability spaces and they are modeled stochastically. Also, the flexibility in MSD to assign different feature dimensions to each individual space enables us to ignore certain features that can cause singularity problem in modeling. Experimental results obtained in handwritten Chinese character recognition indicate MSD provides 1.3%-2.8% character recognition accuracy improvement across different recognition systems where MSD significantly improves discrimination among confusable characters with similar trajectories.

Free style Chinese handwriting recognition continues to pose a challenge to researchers due to the variety of writing styles. To recognize handwritten characters in an online mode, Hidden Markov Model (HMM) has been naturally adopted to model the pen trajectory of a character and a decent recognition performance is achieved. In this study, we start from a maximum likelihood trained HMM model and focus on minimizing recognition errors at the radical (sub- character) level to optimize the recognition performance. A novel Minimum Radical Error discriminative training criterion is proposed, and compared with the discrimination at the character level, our new approach further reduces the character errors by 15.6% relatively (29.0% overall reduction from the maximum likelihood baseline model) on a Chinese database.

This paper proposes a new approach for measuring the target cost in unit selection, where the difference between the target and candidate units is estimated by the Kullback-Leibler divergence (KLD) between the context-dependent hidden Markov models (HMM). In order to model the left/right phonetic context, biphone models are generated by merging regular tri-phone HMMs sharing the same left/right phonetic context. To characterize prosodic contexts, various sets of prosody-sensitive monophone HMMs are trained. KLDs between these context models are calculated as the replacement cost between the contexts. Perceptual experiments show that the resulting synthesized speech sounds slightly better than those with the manually-tuned costs. An important advantage is that the proposed method can be conveniently applied to new corpora or languages without the need of collecting perceptual data

We propose a convenient handwriting user interface for correcting speech recognition errors efficiently. Via the proposed hand-marked correction on the displayed recognition result, substitution, deletion and insertion errors can be corrected efficiently by rescoring the word graph generated in the recognition pass. A new path in the graph that matches the user's feedback in the maximum likelihood sense is found.With the aid of language model and hand corrections part in the best decoded path, rescoring the word graph can correct more errors than user provides. All recognition errors can be corrected after finite number of corrections. Experimental results show that by indicating one word error in user feedback, 33.8% of the erroneous sentences can be corrected; while by indicating one character error, 12.9% of the erroneous sentences can be corrected.

We propose to use Minimum Divergence(MD) as a new measure of errors in discriminative training. To focus on improving discrimination between any two given acoustic models, we refine the error definition in terms of Kullback-Leibler Divergence (KLD) between them. The new measure can be regarded as a modified version of Minimum Phone Error (MPE) but with a higher resolution than just a symbol matching based criterion. Experimental recognition results show the new MD based training yields relative word error rate reductions of 57.8% and 6.1% on TIDigits and Switchboard databases, respectively, in comparing with the ML trained baseline systems. The recognition performance of MD is also shown to be consistently better than that of MPE.

Word posterior probability (WPP) computed over LVCSR word graphs has been used successfully in measuring confidence of speech recognition output. However, for certain applications the word graph is too sparse to warrant reliable WPP estimation. In this paper, we incorporate subword units as background models to generate a subword graph for estimating posterior probability. Experiments on both English and Chinese databases show that syllable background models can repopulate the dynamic hypothesis space for effective computation of confidence measure. The resultant posterior probability confidence measure achieves 94.3% and 95.2% Out-Of-Vocabulary (OOV) word detection / rejection in English and Chinese, respectively. Correspondingly, confidence error rates are at 6.0% and 6.4%, respectively.