Recent advances in deep learning and automatic speech recognition have improved the accuracy of end-to-end speech recognition systems, but recognition of personal content such as contact names remains a challenge. In this work, we describe our personalization solution for an end-to-end speech recognition system based on connectionist temporal classification. Building on previous work, we present a novel method for generating additional subword tokenizations for personal entities from their pronunciations. We show that using this technique in combination with two established techniques, contextual biasing and wordpiece prior normalization, we are able to achieve personal named entity accuracy on par with a competitive hybrid system.

Recent advances in deep learning and automatic speech recognition (ASR) have enabled the end-to-end (E2E) ASR system and boosted the accuracy to a new level. The E2E systems implicitly model all conventional ASR components, such as the acoustic model (AM) and the language model (LM), in a single network trained on audio-text pairs. Despite this simpler system architecture, fusing a separate LM, trained exclusively on text corpora, into the E2E system has proven to be beneficial. However, the application of LM fusion presents certain drawbacks, such as its inability to address the domain mismatch issue inherent to the internal AM. Drawing inspiration from the concept of LM fusion, we propose the integration of an external AM into the E2E system to better address the domain mismatch. By implementing this novel approach, we have achieved a significant reduction in the word error rate, with an impressive drop of up to 14.3% across varied test sets. We also discovered that this AM fusion approach is particularly beneficial in enhancing named entity recognition.

When producing speech-to-text (STT) systems on a lower resource language, it is often beneficial to use knowledge obtained from a significantly larger multilingual dataset. We have seen benefits from using a multilingual TDNN as initialization for training an acoustic model on a target low resource language. In this work, we expand upon recent research that found benefits from applying sequential low-rank factorization (LRF) by extending it to a TDNN acoustic model trained on a large multilingual corpus. We also examine and optimize the knowledge transfer methodology, with the goal of avoiding the loss of useful information from the multilingual initialization during the knowledge transfer process. Our approach limits the updates to the multilingual network parameters during lattice-free maximum mutual information (LF-MMI) training on the target low resource language by fixing the multilingual network parameters and only optimizing the target output layer. The multilingual parameters and new output layer are jointly optimized using the state-level minimum Bayes risk (sMBR) objective function. By combining sequential LRF with this optimization method, we show across low resource target languages an average absolute WER reduction of 1.2%, yielding a better result than our previous best approach.

In this paper, we continue our work on linear least squares based adaptation (LLS) for deep neural networks. We show that our previously proposed algorithm is a special case of an optimization algorithm called Alternating Direction Method of Multipliers (ADMM). We demonstrate that the adaptation algorithm can improve the performance on various deep neural networks including the bidirectional long short term memory (BLSTM). On the Switchboard subset of the Hub5 2000 evaluation set, we show that LLS adaptation can achieve 6 to 9% relative word error rate (WER) reduction, and improve our two-pass system to 7.5% WER. In this paper, we also analyze the factors that could contribute to the success of an adaptation algorithm. This helps us to understand under what circumstances, adaptation could improve the system performance.

We present an importance sampling based approach to the active learning problem of selecting additional training data to supplement a seed model. Our proposed Δ-AUC selection optimizes AUC improvement in keyword search and is evaluated on the Spanish Fisher corpus. We show that over different training data sizes, Δ-AUC selection consistently outperforms random sampling by 1.05% to 2.69% absolute AUC and requires no more than 60% of the transcriptions needed by random sampling to achieve the same AUC. On terms not seen in the original seed model training, the proposed algorithm achieves a 3.47% better AUC and 4.66% reduction in word error rate. We also introduce a regression analysis model that can refine our Δ-AUC strategy in the future.

Recent work reported on fast speaker search over large speech data corpora has focused on using locality sensitive hashing (LSH) search with hashing functions approximating i-vector based cosine distances (CosDist) for model comparisons. Because of the superior performance of probabilistic linear discriminant analysis (PLDA) model reported on speaker identification (SID) in recent years, in this paper we focus on using PLDA for fast speaker search. It is challenging to approximate PLDA well with simple hashing functions, resulting in difficulty to combine it with LSH search. As an alternative, we adopt a clustering-based pruning strategy to speed up PLDA search. Our results show the strategy can significantly speed up search with minimal performance loss. Another focus of this work is on PLDA model adaptation to mismatched conditions under which the fast search runs. The technique we adopt to adapt the PLDA model is based on the LDA adaptation method reported in [1], primarily adapting the LDA transform. Our results show this adaptation improves PLDA performance significantly (over 25% relative) on data collected in different conditions. Our speed-up experiments running with adapted LDA show that gains from the adapted PLDA are retained after the speed-up.

We present a Mandarin keyword search system that uses a large vocabulary recognizer to generate consensus networks at various resolutions: word, character, syllable and phone. In order to achieve fast and accurate search, we propose the use of an efficient approximate-match dynamic programming algorithm that finds the best alignment between the target query and the consensus network. Experiments with Mandarin conversational telephone speech show that the approximate-match search improves detection accuracy by more than 10% for rare words that are not present in the recognizer's dictionary (OOV terms). We also found OOV terms to benefit most from system combination, where we observe a roughly 10% improvement relative to the best single system.

Using speaker adaptation parameters, such as maximum likelihood linear regression (MLLR) adaptation matrices, as features for speaker recognition (SR) has been shown to perform well and can also provide complementary information for fusion with other acoustic-based SR systems, such as GMM-based systems. In order to estimate the adaptation parameters, a speech recognizer in the SR domain is required which in turn requires transcribed training data for recognizer training. This limits the approach only to domains where training transcriptions are available. To generalize the adaptation parameter approach to domains without transcriptions, we propose the use of self-organized unit recognizers that can be trained without supervision (or transcribed data). We report results on the 2002 NIST speaker recognition evaluation (SRE2002) extended data set and show that using MLLR parameters estimated from SOU recognizers give comparable performance to systems using a matched recognizers. SOU recognizers also outperform those using cross-lingual recognizers. When we fused the SOU- and word recognizers, SR equal error rate (EER) can be reduced by another 15%. This suggests SOU recognizers can be useful whether or not transcribed data for recognition training are available.

This paper explores both supervised and unsupervised topic modeling for spoken audio documents using only phonetic information. In cases where word-based recognition is unavailable or infeasible, phonetic information can be used to indirectly learn and capture information provided by topically relevant lexical items. In some situations, a lack of transcribed data can prevent supervised training of a same-language phonetic recognition system. In these cases, phonetic recognition can use cross-language models or self-organizing units (SOUs) learned in a completely unsupervised fashion. This paper presents recent improvements in topic modeling using only phonetic information. We present new results using recently developed techniques for discriminative training for topic identification used in conjunction with recent improvements in SOU learning. A preliminary examination of the use of unsupervised latent topic modeling for unsupervised discovery of topics and topically relevant lexical items from phonetic information is also presented.

Named entity (NE) extraction is traditionally applied to written text. Some recent works extend the extraction to broadcast speech but most of these approaches simply cascade a speech-to-text (STT) engine and a named entity tagger without any information sharing between the two. In this paper, we extract named entities from conversational speech and explore approaches to couple the STT and NE extraction beyond a simple cascade. We propose a new approach that adapts the STT language model based on the extracted named entities. This steers the STT to focus on the subset of the vocabulary that is most important to NE extraction. We performed a number of experiments on English conversational speech in the Fisher Corpus with different STT recognition speeds. We show that the language model (LM) adaptation approach increases NE extraction recall and improves NE performance by as much as 15% as measured by F-score, by significantly improving the word error rate (WER) of NE words, with minimal impact on the overall WER.

In language identification and other speech applications, discriminatively trained models often outperform nondiscriminative models trained with the maximum-likelihood criterion. For instance, discriminative Gaussian mixture models (GMMs) are typically trained by optimizing some discriminative criteria that can be computationally expensive and complex to implement. In this paper, we explore a novel approach to discriminative GMM training by using a variant the boosting framework (R. Schapire, ldquoThe boosting approach to machine learning, an overview,rdquoProc.MSRIWorkshoponNonlinearEstimationandClassification,2002) from machine learning, in which an ensemble of GMMs is trained sequentially. We have extended the purview of boosting to class conditional models (as opposed to discriminative models such as classification trees). The effectiveness of our boosting variation comes from the emphasis on working with the misclassified data to achieve discriminatively trained models. Our variant of boosting also includes utilizing low confidence data classifications as well as misclassified examples in classifier generation. We further apply our boosting approach to anti-models to achieve additional performance gains. We have applied our discriminative training approach to a variety of language identification experiments using the 12-language NIST 2003 language identification task. We show the significant performance improvements that can be obtained. The experiments include both acoustic as well as token-based speech models. Our best performing boosted GMM-based system on the 12-language verification task has a 2.3% EER.

N-best or lattice-based tokenization has been widely used in speech-related classification tasks. In this paper, we extended the n-best tokenization approach to GMM-based language identification systems with either maximum likelihood (ML) trained or SVM-based language models. We explored the effect of n-best tokenization in training or testing, and its interaction with n-gram order and system fusion. We showed that for both systems, the n-best tokenization gives good performance improvement. However, the SVM-based system benefited from both n-best training and test while the ML-trained system can only benefit from n-best training. Results show n-best tokenization can reduce the relative EER of our best GMM-SVM system by about 5% for 30s and 10s tests.

One of the most common approaches in language verification (LV) is the phonotactic language verification. Currently, LV performances for different languages under different environments and durations have to be compared experimentally and this can make it difficult to understand LV performances across corpora or durations. LV can be viewed as a special case of hypothesis testing such that Neyman-Pearson theorem and other information theoretic analysis are applicable. In this paper, we introduce a measure of phonotactic confusablity based on the phonotactic distribution, and make it possible to assess the difficulty of the verification problem analytically. We then propose a method of predicting LV performance. The effectiveness of the proposed approach is demonstrated on the NIST 2003 language recognition evaluation test set.

In this paper, we adopt the boosting framework to improve the performance of acoustic-based Gaussian mixture model (GMM) Language Identification (LID) systems. We introduce a set of low-complexity, boosted target and anti-models that are estimated from training data to improve class separation, and these models are integrated during the LID backend process. This results in a fast estimation process. Experiments were performed on the 12-language, NIST 2003 language recognition evaluation classification task using a GMM-acoustic-score- only LID system, as well as the one that combines GMM acoustic scores with sequence language model scores from GMM tokenization. Classification errors were reduced from 18.8% to 10.5% on the acoustic-score-only system, and from 11.3% to 7.8% on the combined acoustic and tokenization system.

Speaker adaptation has been widely used in speech recognition. With small amount of adaptation data, Reference Speaker Weighting (RSW) adaptation was previously proposed for fast HMM adaptation, and has been shown to outperform the more commonly used maximum likelihood linear regression (MLLR) adaptation. Extending our previous work [1,2] of applying the Polynomial Segment Models (PSMs) in large vocabulary continuous speech recognition (LVCSR) on the WSJ Nov 92 evaluation, we derive the PSM-based RSW fast adaptation technique in this paper. Different from the HMMs, in which the model means are constants within a state, the PSM means are curves represented by polynomials. Experimental results showed that the PSM-based RSW gave approximately the same relative improvement over the unadapted model as in the HMM case. Comparing the PSM-based RSW and MLLR, the PSM-based RSW is more powerful when the amount of adaptation data available is limited. However, it could quickly saturate with increase in adaptation data.

Robustness has been an important issue for applying speech technologies to real applications. While the Polynomial Segment Models (PSMs) have been shown to outperform HMM under the clean environment, the segmental likelihood evaluation may make the PSM distributions sharper and may adversely affect their performance in mis-matched conditions. In this paper, we explore the robustness properties of the PSM under noisy and channel mis-match conditions. In addition, unsupervised adaptation techniques have been shown to work well for environmental adaptation even with small amount of adaptation data. Thus, it is interesting to compare the PSMs' and the HMMs' performances after applying two types of unsupervised adaptation: the Maximum Likelihood Linear Regression (MLLR) and the Reference Speaker Weighting (RSW). Experiments were performed on the Aurora 4 corpus under both clean and multi-conditional training. Our results show that even under noisy and mis-match conditions, the PSMs performed well compared to the HMMs both before and after environmental adaptation. Using the best lattice, the RSW adapted PSM gave word error rates of 26.5% and 21.3% for clean and multi-conditional training respectively which were approximately 24% better than the unadapted HMM.

Tone has a crucial role in Mandarin speech in distinguishing ambiguous words. Most state-of-the-art Mandarin automatic speech recognition systems adopt embedded tone modeling, where tonal acoustic units are used and F0 features are appended to the spectral feature vector. In this paper, we combine the embedded aproach (using improved F0 smoothing) with explicit tone modeling in rescoring the output lattices. Oracle experiments indicate 32% relative improvement can be achieved by rescoring with perfect tone information. Recognition experiments on Mandarin broad-cast news show that, even with an accuracy of only 70%, the explicit tone classifier offers complementary knowledge and improves performance significantly. Through the combination of tone modeling techniques, the character error rate on the CTV test set can be improved from 13.0% to 11.5%.

Automatic language identification (LID) decisions are made based on scores of language models (LM). In our previous paper [1], we have shown that replacing n-gram LMs with SVMs significantly improved performance of both the PPRLM and GMM-tokenization-based LID systems when tested on the OGI-TS corpus. However, the relatively small corpus size may limit the general applicability of the findings. In this paper, we extend the SVM-based approach on the larger CallFriend corpus evaluated using the NIST 1996 and 2003 evaluation sets. With more data, we found that SVM is still better than n-gram models. In addition, back-end processing is useful with SVM scores in CallFriend which differs from our observation in the OGI-TS corpus. By combining the SVM-based GMM and phonotactic systems, our LID system attains an ID error of 12.1% on NIST 2003 evaluation set which is more than 4% (25% relatively) better than the baseline n-gram system.

Speaker adaptation has long been applied to improve speech recognition performance of hidden Markov model (HMM)-based systems. Recently, the polynomial segment model (PSM) has been shown as a viable alternative that can significantly improve the performance of large vocabulary continuous speech recognition (LVCSR). In this letter, we extend the widely used HMM-based maximum likelihood linear regression (MLLR) speaker adaptation technique to PSMs. PSM properties, such as using segment as a modeling unit, and a polynomial curve as model mean, are taken into account in deriving the PSM-based MLLR. Experiments show that PSM-based MLLR adaptation performs equally well as the HMM-based MLLR adaptation with about 19% relative improvement from the SI model. In addition, another 5% relative improvement can be obtained by combining the adapted PSMs and HMMs

Traditionally, static mel-frequency cepstral coefficients (MFCCs) are derived by discrete cosine transformation (DCT), and dynamic MFCCs are derived by linear regression. Their derivation may be generalized as a frequency-domain transformation of the log filter-bank energies (FBEs) followed by a time-domain transformation. In the past, these two transformations are usually estimated or optimized separately. In this letter, we consider sequences of log FBEs as a set of spectrogram images and investigate an image compression technique to jointly optimize the two transformations so that the reconstruction error of the spectrogram images is minimized; there is an efficient algorithm that solves the optimization problem. The framework allows extension to other optimization costs as well

Since utterance verification (UV) may be treated as a two-class classification problem, it may be improved with discriminative training such as minimum verification error training or minimum verification error rate training. However, since in practice, one usually has to pick a specific false-acceptance or false-rejection rate for one's system, it is more desirable to optimize UV performance at a particular operating point. In this letter, we show that further improvement can be achieved by treating UV at a specific operating point as a constrained optimization problem

Polynomial segment models (PSMs), which are generalization of the hidden Markov models (HMMs), have opened an alternative research direction for speech recognition. However, they have been limited by their computational complexity. Traditionally, any change in PSM segment boundary requires likelihood recomputation of all the frames within the segment. This makes the PSM's segment likelihood evaluation an order of magnitude more expensive than the HMM's. Furthermore, because recognition using segment models needs to search over all possible segment boundaries, the PSM recognition is computationally unfeasible beyond N-best rescoring. By exploiting the properties of the time normalization in PSM, and by decomposing the PSM segment likelihood into a simple function of "sufficient statistics", in this paper, we show that segment likelihood can be evaluated efficiently in an order of computational complexity similar to HMM. In addition, by reformulating the PSM recognition as a search for the optimal path through a graph, this paper introduces a fast PSM search algorithm that intelligently prunes the number of hypothesized segment boundaries, such that PSM recognition can be performed in an order of complexity similar to HMM. We demonstrate the effectiveness of the proposed algorithms with experiments using a PSM-based recognition system on two different recognition tasks: TIDIGIT digit recognition and the Wall Street Journal dictation task. In both tasks, PSM recognition is feasible and out-performed traditional HMM by more than 14%.

The Viterbi algorithm has been successfully applied to different pattern recognition and communication tasks. However, if some observations are corrupted by unknown impulsives noise which are not accounted for by the distortion measures, recognition performance can degrade significantly. In this paper, we propose a robust Viterbi algorithm to handle short impulsive noises with unknown characteristics by means of joint decoding and detection during the Viterbi search. To make the algorithm applicable to different noisy conditions with varying amounts of impulsive noise, we further proposed an approach to efficiently estimate the number of corruptions. We demonstrate the effectiveness of the proposed robust algorithms using spoken digit recognition experiments under two different impulsive noise environments. Under random Gaussian replacement noise, the proposed algorithm reduced digit error by more than 65%. Under the GSM network environment in which lost frames are replaced by interpolated neighboring frames, the robust algorithm reduced digit error by 20%. Furthermore, the proposed algorithm does not degrade performance when impulsive noise is not present

The polynomial segment model (PSM) has opened up an alternative research direction for acoustic modeling. In our previous papers, we proposed efficient incremental likelihood evaluation and EM training algorithms for PSM, that significantly improve the speed of PSM training and recognition. In this paper, we shift our focus to use PSM on large vocabulary recognition. Recognition via N-best re-scoring shows that PSM models out-performed HMM on the 5 k closed vocabulary Wall Street Journal Nov 92 testset. Our best PSM model achieved 7.15% WER compare with 7.81% using 16 mixture HMM model. Specifically, we used sub-phonetic PSM that represents a phoneme as multiple independent segmental units that allows for more effective model sharing. Also, we derived and compared different top-down mixture growing approaches that are orders of magnitude more efficient than previously proposed bottom-up agglomerative clustering techniques. Experimental results show that the top-down clustering performs better than the bottom-up approaches.

The paper proposes an articulatory feature-based conditional pronunciation modeling (AFCPM) technique for speaker verification. The technique captures the pronunciation characteristics of speakers by modeling the linkage between the actual phones produced by the speakers and the state of articulations during speech production. The speaker models, which consist of conditional probabilities of two articulatory classes, are adapted from a set of universal background models (UBMs) via MAP adaptation. This creates a direct coupling between the speaker and background models, which prevents over-fitting the speaker models when the amount of speaker data is limited. Experimental results demonstrate that MAP adaptation not only enhances the discriminative power of the speaker models but also improves their robustness against handset mismatches. Results also show that fusing the scores derived from an AFCPM-based system and a conventional spectral-based system achieves an error rate that is significantly lower than that which can be achieved by the individual systems. This suggests that AFCPM and spectral features are complementary to each other.

Lack of data is a problem in training language models for conversational speech recognition, particularly for languages other than English. Experiments in English have successfully used Web-based text collection, targeted for a conversational style, to augment small sets of transcribed speech; we look at extending these techniques to Mandarin. In addition, we investigate different techniques for topic adaptation. Experiments in recognizing Mandarin telephone conversations show that the use of filtered Web data leads to a 28% reduction in perplexity and 7% reduction in character error rate, with most of the gain due to the general filtered Web data.

Articulatory feature-based conditional pronunciation modeling (AFCPM) aims to capture the pronunciation characteristics of speakers by modeling the linkage between the states of articulation during speech production and the actual phones produced by a speaker. Previous AFCPM systems use one discrete density function for each phoneme to model the pronunciation characteristics of speakers. This paper proposes using a mixture of discrete density functions for AFCPM. In particular, the pronunciation characteristics of each phoneme is modeled by two density functions: one responsible for describing the articulatory features that are more relevant to vowels and the other for consonants. Verification scores are the weighted sum of the outputs of the two models. To enhance the resolution of the pronunciation models, four articulatory properties (front-back, liprounding, place of articulation, and manner of articulation) are used for pronunciation modeling. The proposed AFCPM is applied to a speaker verification task. Results show that using four articulatory features achieves a lower error rate as compared to using two features (manner and place of articulation) only. It was also found that dividing the articulatory properties into two groups is an effective means of solving the data-sparseness problem encountered in the training phase of AFCPM systems. 1.

With the advance in semiconductor memory and the availability of very large speech corpora (of hundreds to thousands of hours of speech), we would like to revisit the use of discrete hidden Markov model (DHMM) in automatic speech recognition. To estimate the discrete density in a DHMM state, the acoustic space is divided into bins and one simply count the relative amount of observations falling into each bin. With a very large speech corpus, we believe that the number of bins may be greatly increased to get a much higher density than before, and we will call the new models, the high-density discrete hidden Markov model (HDDHMM). Our HDDHMM is different from traditional DHMM in two aspects: firstly, the codebook will have a size in thousands or even tens of thousands; secondly, we propose a method based on scalar quantization indexing so that for a d-dimensional acoustic vector, the discrete codeword can be determined in O(d) time. During recognition, the state probability is reduced to an O(1) table look-up. The new HDDHMM was tested on WSJO with 5K vocabulary. Compared with a baseline 4-stream continuous density HMM system which has a WER of 9.71 %, a 4-stream HDDHMM system converted from the former achieves a WER of 11.60%, with no distance or Gaussian computation.

How important is tone to Mandarin speech recognition? Because contextual information can resolve some lexical ambiguities, the question has been raised as to whether a strong language model obviates the need for explicit tone modeling. This letter addresses that question by evaluating the effect of tone information on word perplexity under different conditions, including knowledge of the correct tone sequence, perfect acoustic information, and imperfect syllable and tone information. Results show that conditioning on tone information can reduce word uncertainty in conversational speech by 11%-20%, depending on the accuracy of syllable and tone recognition.

Since the discriminative powers of different phones vary, weighting techniques can be applied to bias the log-likelihood ratio of different phone segments to emphasize the more discriminating phones. First, we derive a Kullback-Leibler distance-based phone log-likelihood weighting scheme that incorporates the duration information. Second, we propose a posterior probability transformation method such that contributions from different phones are weighted implicitly according to their discrimination power. Using the two proposed approaches reduces speaker verification equal error rate by more than 10% on the YOHO database.

It is well known that spectrogram readers can classify different phones from their spectral-time characteristics, such as the formants. In this paper we present a novel acoustic model for phone classification based on the implicit estimation of the spectral peak trajectory as a polynomial time function. By making use of the known relationship between the spectral peak information and the cepstral coefficients, cepstral-based phone trajectories are built as functions of the hidden spectral trajectories. This captures the intuitive formant trajectories in the spectral domain while allowing speech modeling to be done in the more familiar cepstral domain. We have evaluated this hidden spectral peak trajectory model in both vowel classification and phone classification tasks. On a simple single Gaussian model, the hidden spectral peak trajectory model outperforms the HMM on both vowel and phone classification tasks. The new method can also be combined with the HMM model. This combination performs better than a more complex HMM with similar number of parameters.

One of the difficulties in using the polynomial segment model (PSM) to capture the temporal correlations within a phonetic segment is the lack of an efficient training algorithm comparable with the Baum-Welch algorithm in HMM. In our previous paper, we introduced a recursive likelihood computation algorithm for PSM recognition which can perform Viterbi-style training. In this paper, we extend the recursive likelihood computation into a fast forward-backward PSM training algorithm that maximizes PSM likelihood. In addition, we introduce an improved PSM, dynamic multi-region PSM, that allows a data-driven alignment between observations and the segment trajectory. The dynamic multi-region PSM model outperforms HMM and traditional PSM in both phone classification and phone recognition tasks on the TIMIT corpus.

Because of the tonal nature of Chinese languages, correct recognition of lexical tones is necessary for Chinese speech recognition. In order to incorporate tone information into Chinese speech recognition, three issues need to be addressed: (i) the representation of the syllable pitch contour as well as the tone contour; (ii) the lexical tone probability estimation; and (iii) the integration of tone probabilities into the Viterbi recognition process. In this paper we propose a robust polynomial segmental representation of the pitch contour coupled with a decision tree based tone classifier. We also propose a novel approach of integrating the decision tree tone classifier directly into a single pass recognition process. The proposed approaches were evaluated on tasks of tone classification and tonal-syllable recognition. In regard to tone classification, the robust decision tree gave a tone classification-accuracy of 89% for isolated syllables and 71.2% for the continuous speech. Moreover, by incorporating the decision tree tone classifier into the Viterbi search, the tonal-syllable recognition error rate in continuous speech was reduced by 13.54%.

This paper proposes a spectral subtraction based speech enhancement algorithm that improves computer based speech recognition. Speech recognition can not be improved by traditional spectral subtraction techniques because of the associated artifacts, such as musical noise. This paper proposes a tonal noise suppression algorithm which can he used for accurate psychoacoustic model parameter estimation. Simulation results show that the spectral subtraction based speech enhancement algorithm with psychoacoustic masking produced by the estimated parameters produces high quality clean speech, which can be used as preprocessing tools to enhance the speech recognition rate. Simulation results using Aurora 3 database are presented.

The introduction of Aurora 4 tasks provides a standard database and methodology for comparing the effectiveness of different robust algorithms on LVCSR. One important issue on Aurora 4 tasks is the computation time involved in evaluating different test conditions. In this paper we show that by employing HTK as the recognition frontend and backend on Aurora 4 tasks with the use of cepstral mean subtraction, 14% relative improvement is achieved on the baseline clean train tasks at a 82.5% time reduction in training time and 40% time reduction on decoding. Furthermore, we found that optimizing the model complexity can increase the recognition performance (in both computation time and accuracy). Accuracy can be further improved with the use of unsupervised MLLR adaptation on one or multiple sentences. The adaptation results show that most of the gain from adaptation comes from adapting to the environment instead of to the speaker. With the use of adaptation,the error rate is reduced from the baseline result of 69.6% to 40%.

Robustness is an important issue in automatic speech recognition systems. When the testing conditions do not match the training condition or when there is insufficient training data, the performance of a system trained by maximum likelihood criterion may degrade significantly. Different robust algorithms were proposed especially for cases in which the mismatch condition is known or can be estimated from the test data. In many practical cases, however, the mismatch information may not be available. In this paper, we propose a robust training algorithm that does not make any assumption about the mismatch condition. Instead, it is based on a neighborhood concept that appropriately broaden the model distributions to increase the model robustness. The proposed algorithm is evaluated in the Aurora3 tasks which shows that the neighborhood approach can reduce the degradation in mismatch conditions.

In the recent automatic language identification research, phonotactic approach has been studied in which all training utterances are passed through a tokenizer in order to get phonetic sequences to train the language model of different languages. The true transcription of the utterances was totally ignored. However, information in the transcription may possess important discriminating power for language identification. In this paper, we propose to use discrete hidden Markov model that takes account of the potential error patterns of the acoustic tokenizer and incorporates the transcription of the utterances in the language model training. Furthermore, with the DHMM approach, LID using multiple phonetic tokenizers can simply be considered as using a multi-dimensional features to the DHMM allowing the making of joint decision earlier in the process. A system employing this approach produces 59.00% and 68.33% accuracy on 10-sec and 45-sec speech respectively on recognizing a close set of six languages in the OGI telephone speech corpus while the phonotactic approach gives 57.00% and 77.50% identification accuracy on 10-sec and 45-sec speech when the phone recognizer uses three-state and three-mixture HMM.

Confidence measures are used in a number of applications to verify the user input or to measure the certainty of the recognition outputs. Most of the HMM-based systems use MFCC features with Gaussian mixtures models to estimate confidence. We propose a new approach to estimate confidence by combining the posterior probabilities of articulatory features (AF) computed by a set of AF classifiers. This AF-based confidence measure gives comparable performance in terms of classification equal error rate (EER) to the Gaussian mixture-based approach but reduces the computation by 50% (as measured by the approximated number of multiplications) and consumes smaller memory. When the AF-based confidence is combined with confidence from the Gaussian mixtures, the EER is further reduced. This AF confidence can be particularly useful for platforms with limited computing resources such as hand-held devices.

The polynomial segment model (PSM), which was first proposed in Gish et al. (1993) and subsequently studied by other researchers, has opened up an alternative research direction for speech recognition. In PSM, speech frames within a segment are jointly modeled such that any change in the boundaries of a segment would require the re-computation of the likelihood of all the frames within the segment. While estimation of the best segment boundaries are possible, the computation consideration typically constrains the PSM model to limit the search to center around some pre-segmentation typically obtained by using another model such as an HMM, in effect limiting the possibility of using PSM itself. In this paper we introduce a new approach to evaluate the likelihood of a PSM segment by efficiently "accumulating" segment likelihood incrementally, i.e. one frame at a time. Based on this incremental likelihood evaluation, an efficient PSM search and training algorithm are also introduced. We show the effectiveness of the incremental likelihood evaluation by building a PSM-based TIMIT recognition system (both training and test) without the need of using another model for pre-segmentation.

One important issue in speech recognition is the ability to handle variations caused by unseen speakers or noise. One approach to handle variations is to adapt the trained acoustic models to the testing environment. Maximum likelihood linear regression is one of these techniques for handling speaker variations. Can speaker adaptation techniques be applicable to handling noise variations? To answer this question, an in-depth analysis on the effects of speaker and noise variations on speech has been performed. It shows that the speaker variation leads to a shift and a rotation in the cepstral features while additive noise causes a shift in the cepstral features and a reduction in the feature variance. This finding is consistent with the recently proposed feature vector normalization technique, which uses the utterance mean and variance to compensate the shift and dynamic range reduction. This empirical normalization has been shown to improve the recognition performance under additive noise and channel effect. However, the estimates may not be optimal. Instead, normalization that maximizes the likelihood is proposed. In addition, we propose two different approximations with different computational complexities. Because the effect of additive noise is dependent on the signal-to-noise ratio, its influence on different phonetic units can vary. We generalized our framework allowing different normalization factors for different phonetic classes. Applying the ML normalization on the Aurora 3 corpus, we obtained a relative improvement of 10% compared to the ESTI standard baseline. Frame skipping Viterbi algorithm (FSVA) has been recently shown to be useful in handling impulsive noises. However, how to handle impulsive noise and additive noise together has not been addressed. As an extension to the ML normalization, we integrate the FSVA algorithm into the ML normalization framework. This integration results in a relative improvement of 25% when impulsive noises are present.

In this paper, we propose two improvements to the articulatory feature (AF) models. We introduce the use of 3-state AF model with multiple observation distributions that gives a better modeling of the articulatory features within a phone. This results in an improvement of about 1% in phone recognition on the TIMIT task. Combining the AF model with acoustic-based HMM achieves an improvement of 1.6% compares to use acoustic features only. We then introduce the asynchronous state combination of the 3-state AF models with acoustic-based HMM and obtain an additional improvement of 1.7%.

Chinese is a tonal language that uses fundamental frequency, in addition to phones for word differentiation. Commonly used front-end features, such as mel-frequency cepstral coefficients (MFCC), however, are optimized for non-tonal languages such as English and are not mainly focused on pitch information that is important for tone identification. In this paper, we examine the integration of tone-related acoustic features for Chinese recognition. We propose the use of the cepstrum method (CEP), which uses the same configurations as in MFCC extraction for the extraction of pitch-related features. The pitch periods extracted from the CEP algorithm can be used directly for speech recognition and do not require any special treatment for unvoiced frames. In addition, we explore a number of feature transformations and find that the addition of a properly normalized and transformed set of pitch related-features can reduce the recognition error rate from 34.61% to 29.45% on the Chinese 1998 National Performance Assessment (Project 863) corpus.

Recognizing speech transmitted over mobile or computer networks poses new challenges such as packet loss in transmission. Viterbi algorithm, the most common speech recognition approach, searches for the most likely state sequence that explains all observation. However, because it implicitly sums the log observation probabilities, the resulting solution is sensitive to outlier frames. In this paper, we propose a robust approach that searches the state sequence that best explains percent of the observation and is insensitive to the corruption of a limited number of frames. We evaluated the proposed algorithm on the TIdigits task. With 10% of the data loss, the proposed algorithm achieves improvement of WER 71.6% for isolated digit recognition and 32.2% for connected digit recognition.

Focuses on the learning of multi-word lexical units, or phrases, and how to model them within the variable n-gram framework. We introduce the notion of context-dependent phrases and suggest an algorithm for the unsupervised learning of phrases. Also, we propose an approach to integrate a phrase grammar and a variable n-gram without the need to explicitly handle multi-word lexical items. The combined variable n-gram phrase grammar improves recognition accuracy on the Switchboard corpus over both the baseline trigram and using a variable n-gram alone.

The use of context-dependent phonetic units together with Gaussian mixture models allows modern-day speech recognizer to build very complex and accurate acoustic models. However, because of data sparseness issue, some sharing of data across dierent triphone states is necessary. The acoustic model design is typically done in two stages, namely, designing the state-tying map and growing the number of mixtures in each tied-state. In the design of the state-tying map, single Gaussians are used to represent the data, ignoring the fact that a single Gaussian is an insucient model. In this paper, we propose a simple modication to the two-stage process by adding a third stage. In this added stage, the state-tying tree is pruned and the pruning is based on the mixture representation of the tied-states. We propose using Bayesian Information Criterion(BIC) as the criterion for this pruning and show that by adding this step, the resulting model is more compact and gives better recognition accura...

Recent progress in variable n-gram language modeling provides an efficient representation of n-gram models and makes training of higher order n grams possible. We apply the variable n-gram design algorithm to conversational speech, extending the algorithm to learn skips and context-dependent classes to handle conversational speech characteristics such as filler words, repetitions, and other disfluencies. Experiments show that using the extended variable n-gram results in a language model that captures 4-gram context with less than half the parameters of a standard trigram while also improving the test perplexity and recognition accuracy.

This paper describes the improvements that resulted in the 1998 Byblos large vocabulary conversational speech recognition (LVCSR) system. Salient among these improvements are: improved signal processing, improved hidden Markov model (HMM) topology, use of quinphone context, introduction of diagonal speaker adapted training (DSAT), incorporation of variance adaptation in the MLLR framework, improvements in language modeling, increase in lexicon size and combination of multiple systems. These changes resulted in about a 7% absolute reduction in word error rates on a balanced Switchboard/Callhome English test set.

This paper presents an approach to routing telephone calls automatically, based upon their speech content. The data consist of a set of calls collected from a customer-service center with a two-level menu, which allows jumping past the second level, and we view the routing of these calls as a topic identification problem. Our topic identifier employs a multinomial model for keyword occurrences. We describe the call routing task in detail, discuss the multinomial model, and present experiments which investigate several issues that arise from using the model for this task.

How to train a speech recognizer with limited amount of training data is of interest to many researcher. We describe how we use BBN's Byblos large vocabulary continuous speech recognition (LVCSR) system for the military air-traffic-control domain where we have less than an hour of training data. We investigate three ways to deal with the limited training data: (1) re-configure the LVCSR system to use fewer parameters, (2) incorporate out-of-domain data, and, (3) use pragmatic information, such as speaker identity and controller function to improve recognition performance. We compare the LVCSR performance to that of the tied-mixture recognizer that is designed for a limited vocabulary. We show that the reconfigured LVCSR system outperforms the tied-mixture system by 10% in absolute word error rate. When enough data is available per speaker, vocal tract length normalization and supervised adaptation techniques can further improve performance by 6% even for this domain with limited training. We also show that the use of out-of-domain data and pragmatic information, if available, can each further improve performance by 1-3%.

This paper presents the 1997 BBN Byblos large vocabulary speech recognition (LVCSR) system. We give an outline of the algorithms and procedures used to train the system, describe the recognizer configuration and present the major technological innovations that lead to the performance improvements. The major testbed we present our results for is the Switchboard corpus, where current word error rates vary from 27% to 34% depending on the test set. In addition, we present results on the CallHome Spanish and Arabic tests, where we demonstrate that technology developed on English corpora is very much portable to other problems and languages.

Parametric trajectory models explicitly represent the tem-poral evolution of the speech features as a Gaussian process with time-varying parameters. HMMs are a special case of such models, one in which the trajectory constraints in the speech segment are ignored by the assumption of condi-tional independence across frames within the segment. In this paper, we investigate in detail some extensions to our trajectory modeling approach aimed at improving LVCSR performance: (i) improved modeling of mixtures of trajec-tories via better initialization, (ii) modeling of context de-pendence, and (iii) improved segment boundaries by means of search. We will present results in terms of both phone classi cation and recognition accuracy on the Switchboard corpus. 1.

This paper explores techniques for utilizing untranscribed training data pools to increase the available training data for automatic speech recognition systems. It has been well estab-lished that current speech recognition technology, especially in Large Vocabulary Conversational Speech Recognition (LVCSR), is largely language independent, and that the dominant factor with regards to performance on a certain language is the amount of available training data ([4]). The paper addresses this need for increased training data by presenting ways to use untranscribed acoustic data to increase the training data size and thus improve speech recognition. 1.

Speech recognition of conversational speech is a difficult task. The performance levels on the Switchboard corpus had been in the vicinity of 70% word error rate. In this paper, we describe the results of applying a variety of modifications to our speech recognition system and we show their impact on improving the performance on conversational speech. These modifications include the use of more complex models, trigram language models, and cross-word triphone models. We also show the effect of using additional acoustic training on the recognition performance. Finally, we present an approach to dealing with the abundance of short words, and examine how the variable speaking rate found in conversational speech impacts on the performance. Currently, the level of performance is at the vicinity of 50% error, a significant improvement over recent levels.

It is often the case that one wants to apply a neural network classifier, trained with a particular mix of the training classes, to a situation where the classes occur in a different proportion. The originally trained network will not be appropriate for the new situation. The authors show that only a single weight of the network needs to be modified to accommodate the network to the new situation. The other weights are invariant to the change in mix.>

Jeanrenaud et al. (1993) introduced the notion of event spotting and showed that the detection of events could be approached as a word spotting problem. The present authors first concentrate on the problem of spotting complex units such as grammatic events and describe the three issues that have to be addressed. These issues are: 1) hypothesizing-how to detect an event, 2) scoring-how to generate a consistent score for the event, 3) identification-what word sequence constitutes the particular realization of the event. The authors also discuss ways to evaluate the performance of an event spotter and how to characterize the complexity of an event. They then present two approaches to spot events. The first approach is based on posterior probability scoring using sub-grammars, and the second uses a large vocabulary recognizer. They show that both approaches have comparable performance on an event spotting task using the Switchboard corpus. In the posterior probability scoring approach, however, there is the advantage of being able to choose an operating point.>

A novel system for extracting information from stereotyped voice traffic is described. Off-the-air recordings of commercial air traffic control communications are interpreted in order to identify the flights present and determine the scenario (e.g., takeoff, landing) that they are following. The system combines algorithms from signal segmentation, speaker segregation, speech recognition, natural language parsing, and topic classification into a single system. Initial evaluation of the algorithm on data recorded at Dallas-Fort Worth airport yields performance of 68% detection of flights with 98% precision at an operating point where 76% of the flight identifications are correctly recognized. In tower recording containing both takeoff and landing scenarios, flights are correctly classified as takeoff or landing 94% of the time.>

The authors present a method for segmenting speech waveforms containing several speakers into utterances, each from one individual, and then identifying each utterance as coming from a specific individual or group of individuals. The procedure is unsupervised in that there is no training set, and sequential in that information obtained in early stages of the process is utilized in later stages.>

A method for segregating speech from speakers engaged in dialogs is described. The method, assuming no prior knowledge of the speakers, employs a distance measure between speech segments used in conjunction with a clustering algorithm to perform the segregation. Properties of the distance measure are discussed, and an air traffic control application is described.>