BERT-era question answering systems have recently achieved impressive performance on several question-answering (QA) tasks. These systems are based on representations that have been pre-trained on self-supervised tasks such as word masking and sentence entailment, using massive amounts of data. Nevertheless, additional pre-training closer to the end-task, such as training on synthetic QA pairs, has been shown to improve performance. While recent work has considered augmenting labelled data and leveraging large unlabelled datasets to generate synthetic QA data, directly adapting to target data has received little attention. In this paper we investigate the iterative generation of synthetic QA pairs as a way to realize unsupervised self adaptation. Motivated by the success of the roundtrip consistency method for filtering generated QA pairs, we present iterative generalizations of the approach, which maximize an approximation of a lower bound on the probability of the adaptation data. By adapting on synthetic QA pairs generated on the target data, our method is able to improve QA systems significantly, using an order of magnitude less synthetic data and training computation than existing augmentation approaches.

This paper presents a machine learning Deep Belief Network technique for interpolation and extrapolation of HRTF (Head Related Transfer Function) databases. In this technique, we decouple the stereo pair of HRTFs for each ear. Furthermore, we remove the inter-aural time differences (ITD) and distance attenuation from the recorded HRTF measurements such that the DBNs only interpolate and extrapolate the spectrum filtering of the audio for the two ears. Testing on the CIPIC and SCUT databases produces results of an average log spectral distortion less than 3 dB for all points around the head.

Recent work on end-to-end automatic speech recognition (ASR) has shown that the connectionist temporal classification (CTC) loss can be used to convert acoustics to phone or character sequences.Such systems are used with a dictionary and separately-trained Language Model (LM) to produce word sequences.However, they are not truly end-to-end in the sense of mapping acoustics directly to words without an intermediate phone representation.In this paper, we present the first results employing direct acoustics-to-word CTC models on two well-known public benchmark tasks: Switchboard and Call-Home.These models do not require an LM or even a decoder at run-time and hence recognize speech with minimal complexity.However, due to the large number of word output units, CTC word models require orders of magnitude more data to train reliably compared to traditional systems.We present some techniques to mitigate this issue.Our CTC word model achieves a word error rate of 13.0%/18.8%on the Hub5-2000 Switchboard/CallHome test sets without any LM or decoder compared with 9.6%/16.0%for phone-based CTC with a 4-gram LM.We also present rescoring results on CTC word model lattices to quantify the performance benefits of a LM, and contrast the performance of word and phone CTC models.

State-of-the-art convolutional neural networks (CNNs) typically use a log-mel spectral representation of the speech signal. However, this representation is limited by the spectro-temporal resolution afforded by log-mel filter-banks. A novel technique known as Deep Scattering Spectrum (DSS) addresses this limitation and preserves higher resolution information, while ensuring time warp stability, through the cascaded application of the wavelet-modulus operator. The first order scatter is equivalent to log-mel features and standard CNN modeling techniques can directly be used with these features. However the higher order scatter, which preserves the higher resolution information, presents new challenges in modelling. This paper explores how to effectively use DSS features with CNN acoustic models. Specifically, we identify the effective normalization, neural network topology and regularization techniques to effectively model higher order scatter. The use of these higher order scatter features, in conjunction with CNNs, results in relative improvement of 7% compared to log-mel features on TIMIT, providing a phonetic error rate (PER) of 17.4%, one of the lowest reported PERs to date on this task.

User authentication in the context of a secure transaction needs to be continuously evaluated for the risks associated with the transaction authorization. The situation becomes even more critical when there are regulatory compliance requirements. Need for such systems have grown dramatically with the introduction of smart mobile devices which make it far easier for the user to complete such transaction quickly but with a huge exposure to risk. Biometrics can play a very significant role in addressing such problems as a key indicator of the user identity and thus reducing the risk of fraud. While unimodal biometrics authentication systems are being increasingly experimented by mainstream mobile system manufacturers (e.g., fingerprint in iOS), we explore various opportunities of reducing risk in a multimodal biometrics system. The multimodal system is based on fusion of several biometrics combined with a policy manager. A new biometric modality: chirography which is based on user writing on multi-touch screens using their finger is introduced. Coupling with chirography, we also use two other biometrics: face and voice. Our fusion strategy is based on inter-modality score level fusion that takes into account a voice quality measure. The proposed system has been evaluated on an in-house database that reflects the latest smart mobile devices. On this database, we demonstrate a very high accuracy multi-modal authentication system reaching an EER of 0.1% in an office environment and an EER of 0.5% in challenging noisy environments.

Log-mel filterbank features, which are commonly used features for CNNs, can remove higher-resolution information from the speech signal. A novel technique, known as Deep Scattering Spectrum (DSS), addresses this issue and looks to preserve this information. DSS features have shown promise on TIMIT, both for classification and recognition. In this paper, we extend the use of DSS features for LVCSR tasks. First, we explore the optimal multi-resolution time and frequency scattering operations for LVCSR tasks. Next, we explore techniques to reduce the dimension of the DSS features. We also incorporate speaker adaptation techniques into the DSS features. Results on a 50 and 430 hour English Broadcast News task show that the DSS features provide between a 4-7% relative improvement in WER over log-mel features, within a state-of-the-art CNN framework which incorporates speaker-adaptation and sequence training. Finally, we show that DSS features are similar to multi-resolution log-mel + MFCCs, and similar improvements can be obtained with this representation.

We propose to adapt deep neural network (DNN) acoustic models to a target speaker by supplying speaker identity vectors (i-vectors) as input features to the network in parallel with the regular acoustic features for ASR. For both training and test, the i-vector for a given speaker is concatenated to every frame belonging to that speaker and changes across different speakers. Experimental results on a Switchboard 300 hours corpus show that DNNs trained on speaker independent features and i-vectors achieve a 10% relative improvement in word error rate (WER) over networks trained on speaker independent features only. These networks are comparable in performance to DNNs trained on speaker-adapted features (with VTLN and FMLLR) with the advantage that only one decoding pass is needed. Furthermore, networks trained on speaker-adapted features and i-vectors achieve a 5-6% relative improvement in WER after hessian-free sequence training over networks trained on speaker-adapted features only.

Several methods have been proposed recently for modeling posterior representations derived from local classifiers [1, 2]. In recent work, Sainath et al. have proposed the use of a tied-mixture-based posterior modeling approach [3] to enhance exemplar-based posterior representations for phone recognition tasks. In this work, we conduct a detailed evaluation to determine the effectiveness of this technique on three representative posterior systems. In addition, we propose and evaluate an alternative discriminative formulation of the posterior modeling objective function that seeks to minimize framelevel errors. In experimental evaluations on the TIMIT corpus, we find that posterior modeling results in relative phone error rate (PER) reductions of between 1.1-5.5% across the systems tested. In fact, using Spif-NN [4, 3] posteriors, we are able to achieve a PER of 18.5; to the best of our knowledge, this is the best result reported in the literature to date. minimize framelevel errors.

Restricted Boltzmann Machines (RBM) continue to be a popular methodology to pre-train weights of Deep Belief Networks (DBNs). However, the RBM objective function cannot be maximized directly. Therefore, it is not clear what function to monitor when deciding to stop the training, leading to a challenge in managing the computational costs. The Sparse Encoding Symmetric Machine (SESM) has been suggested as an alternative method for pre-training. By placing a sparseness term on the NN output codebook, SESM allows the objective function to be optimized directly and reliably be monitored as an indicator to stop the training. In this paper, we explore SESM to pre-train DBNs and apply this the first time to speech recognition. First, we provide a detailed analysis comparing the behavior of SESM and RBM. Second, we compare the performance of SESM pre-trained and RBM pre-trained DBNs on TIMIT and a 50 hour English Broadcast News task. Results indicate that pre-trained DBNs using SESM and RBMs achieve comparable performance and outperform randomly initialized DBNs with SESM providing a much easier stopping criterion relative to RBM.

In this paper, we propose a novel exemplar based technique for classification problems where for every new test sample the classification model is re-estimated from a subset of relevant samples of the training data.We formulate the exemplar-based classification paradigm as a sparse representation (SR) problem, and explore the use of convex hull constraints to enforce both regularization and sparsity. Finally, we utilize the Extended Baum-Welch (EBW) optimization technique to solve the SR problem. We explore our proposed methodology on the TIMIT phonetic classification task, showing that our proposed method offers statistically significant improvements over common classification methods, and provides an accuracy of 82.9%, the best single-classifier number reported to date.

We introduce the Line Search A-Function (LSAF) technique that generalizes the Extended-Baum Welch technique in order to provide an effective optimization technique for a broader set of functions. We show how LSAF can be applied to functions of various probability density and distribution functions by demonstrating that these probability functions have an A-function. We also show that sparse representation problems (SR) that use 11 or combination of 11/12 regularization norms can also be efficiently optimized through an A-function derived for their objective functions. We will demonstrate the efficiency of LSAF for SR problems through simulations by comparing it with Approximate Bayesian Compressive Sensing method that we recently applied to speech recognition.

Exemplar-based techniques, such as k-nearest neighbors (kNNs) and Sparse Representations (SRs), can be used to model a test sample from a few training points in a dictionary set. In past work, we have shown that using a SR approach for phonetic classification allows for a higher accuracy than other classification techniques. These phones are the basic units of speech to be recognized. Motivated by this result, we create a new dictionary which is a function of the phonetic labels of the original dictionary. The SR method now selects relevant samples from this new dictionary to create a new feature representation of the test sample, where the new feature is better linked to the actual units to be recognized. We will refer to these new features as Spif. We present results using these new Spiffeatures in a Hidden Markov Model (HMM) framework for speech recognition. We find that the Spiffeatures allow for a 2.9% relative reduction in Phonetic Error Rate (PER) on the TIMIT phonetic recognition task. Furthermore, we find that the Spiffeatures allow for a 4.8% relative improvement in Word Error Rate (WER) on a large vocabulary 50 hour Broadcast News task.

The use of exemplar-based methods, such as support vector machines (SVMs), k-nearest neighbors (kNNs) and sparse representations (SRs), in speech recognition has thus far been limited. Exemplar-based techniques utilize information about individual training examples and are computationally expensive, making it particularly difficult to investigate these methods on large-vocabulary continuous speech recognition (LVCSR) tasks. While research in LVCSR provides a good testbed to tackle real-world speech recognition problems, research in this area suffers from two main drawbacks. First, the overall complexity of an LVCSR system makes error analysis quite difficult. Second, exploring new research ideas on LVCSR tasks involves training and testing state-of-the-art LVCSR systems, which can render a large turnaround time. This makes a small vocabulary task such as TIMIT more appealing. TIMIT provides a phonetically rich and hand-labeled corpus that allows easy insight into new algorithms. However, research ideas explored for small vocabulary tasks do not always provide gains on LVCSR systems. In this paper, we combine the advantages of using both small and large vocabulary tasks by taking well-established techniques used in LVCSR systems and applying them on TIMIT to establish a new baseline. We then utilize these existing LVCSR techniques in creating a novel set of exemplar-based sparse representation (SR) features. Using these existing LVCSR techniques, we achieve a phonetic error rate (PER) of 19.4% on the TIMIT task. The additional use of SR features reduce the PER to 18.6%. We then explore applying the SR features to a large vocabulary Broadcast News task, where we achieve a 0.3% absolute reduction in word error rate (WER).

Compressive sensing (CS) is a popular technique used to reconstruct a signal from few training examples, a problem which arises in many machine learning applications. In this paper, we introduce a technique to guarantee that our data obeys certain isometric properties. In addition, we introduce a bayesian approach to compressive sensing, which we call ABCS, allowing us to obtain complete statistics for estimated parameters. We apply these ideas to fMRI classification and find that by isometrically transforming our data, significant improvements in classification accuracy can be achieved using the LASSO and Dantzig selector methods, two standard techniques used in CS. In addition, applying the ABCS method offers improvements in classification accuracy over both LASSO and Dantzig. Finally, we find that applying both the ABCS method together with isometric transformations, we are able to achieve an error rate of 0.0%.

Sparse representation phone identification features (SPIF) is a recently developed technique to obtain an estimate of phone posterior probabilities conditioned on an acoustic feature vector. In this paper, we explore incorporating SPIF phone posterior probability estimates in large vocabulary continuous speech recognition (LVCSR) task by including them as additional features of exponential densities that model the HMM state emission likelihoods. We compare our proposed approach to a number of other well known methods of combining feature streams or multiple LVCSR systems. Our experiments show that using exponential models to combine features results in a word error rate reduction of 0.5% absolute (18.7% down to 18.2%); this is comparable to best error rate reduction obtained from system combination methods, but without having to build multiple systems or tune the system combination weights.

The use of exemplar-based techniques for both speech classification and recognition tasks has become increasingly popular in recent years. However, the notion of why sparseness is important for exemplar-based speech processing has been relatively unexplored. In addition, little analysis has been done in speech processing on the appropriateness of different types of sparsity regularization constraints. The goal of this paper is to answer the above two questions, both through mathematically analyzing different sparseness methods and also comparing these approaches for phonetic classification in TIMIT.

Sparse representations (SRs) are often used to characterize a test signal using few support training examples, and allow the number of supports to be adapted to the specific signal being categorized. Given the good performance of SRs compared to other classifiers for both image classification and phonetic clas-sification, in this paper, we extended the use of SRs for text classification, a method which has thus far not been explored for this domain. Specifically, we demonstrate how sparse repre-sentations can be used for text classification and how their per-formance varies with the vocabulary size of the documents. In addition, we also show that this method offers promising results over the Naive Bayes (NB) classifier, a standard baseline classi-fier used for text categorization, thus introducing an alternative class of methods for text categorization. 1.

In this paper, we explore the use of exemplar-based sparse representations (SRs) to map test features into the linear span of training examples. We show that the frame classification ac-curacy with these new features is 1.3 % higher than a Gaussian Mixture Model (GMM), showing that not only do SRs move test features closer to training, but also move the features closer to the correct class. Given these new SR features, we train up a Hidden Markov Model (HMM) on these features and per-form recognition. On the TIMIT corpus, we show that apply-ing the SR features on top of our best discriminatively trained system allows for a 0.7 % absolute reduction in phonetic er-ror rate (PER), from 19.9 % to 19.2%. In fact, after applying model adaptation we reduce the PER to 19.0%, the best re-sults on TIMIT to date. Furthermore, on a large vocabulary 50 hour broadcast news task, we achieve a reduction in word error rate (WER) of 0.3 % absolute, demonstrating the benefit of this method for large vocabulary speech recognition. 1.

We demonstrate the generalizability of the Extended Baum-Welch (EBW) algorithm not only for HMM parameter estimation but for decoding as well. We show that there can exist a general function associated with the objective function under EBW that reduces to the well-known auxiliary function used in the Baum-Welch algorithm for maximum likelihood estimates. We generalize representation for the updates of model parameters by making use of a differentiable function (such as arithmetic or geometric mean) on the updated and current model parameters and describe their effect on the learning rate during HMM parameter estimation. Improvements on speech recognition tasks are also presented here. 1.

Summary form only given. Two forces are in pursuit of discovering the possibilities of speech technology automation. First is the global research and development community which has been hard at work for improving the performance and usability of the technology. Second is the business community which constantly evaluates the performance of the technology against the expectation of the user community for delivering solutions such as a spoken car navigation system. While the performance improvement has been on a constant positive progress curve, the market opportunity has been on a much more uncertain curve. For example, the early vision of delivering a dictation solution has been on hold in recent years while it enjoyed enormous interest in the 90 s. At the same time, some industry experts predict that this vision will be fulfilled soon because of the usability needs of billions of mobile devices in use today. Analogies can be drawn for the use of speech technologies for call centers self service interaction. While we have seen a much bigger market success, some industry experts predict that web self services will slow down the use of speech self service. So, where does the truth lie? What are those market opportunities that are clear winners? What opportunities will open up in future and what are their technical challenges? In this talk, we will address some of these questions.

We describe the design and use of linear hyperplanes to partition the feature space of a classifier. The objective of the partitioning is to minimize the average entropy of the class distribution in the final partitions. The hyperplanes are characterized by a vector /spl nu//sub n/ and scalar h/sub n/, which are computed with the objective of maximizing the mutual information associated with the partitioning. We show that the problem of designing the /spl nu//sub n/ can be simplified to an approximately equivalent problem, the solution of which, interestingly enough, turns out to be the linear discriminant of the data. We also describe a decision-tree based technique that partitions the feature space in a hierarchical fashion where each node in the decision tree represents a linear hyperplane that further partitions the feature space into two regions. The end result of the tree-growing process is that the entire feature space is partitioned into nonoverlapping regions where each region is bounded by a number of hyperplanes. As the criterion for the design of the hyperplanes is the minimization of the average class entropy of the regions, each region is characterized by the occurrence of only a small number of classes. The partitioning information provided by the decision-tree can then be used to eliminate a number of classes from being considered, and hence simplifies the job of the classifier. We show the application of the decision-tree as a preprocessor to a classifier in the speech recognition problem. Here the classes are modeled with mixtures of tens to hundreds of thousands of Gaussians, and the use of the partitioning information reduces the computation associated with the classification process by factors larger than 20 with negligible degradation in the word error rate (note, however, that the overall decoding process comprises other steps in addition to the classification-the computational speedup mentioned above does not affect the computation of these other steps). We also compare the performance of this quantization scheme with standard Gaussian clustering schemes and show that the decision-tree based quantization provides better performance in most cases.

A speaker adaptation strategy is described that is based on finding a subset of speakers, from the training set, who are acoustically close to the test speaker, and using only the data from these speakers (rather than the complete training corpus) to reestimate the system parameters. Further, a linear transformation is computed for every one of the selected training speakers to better map the training speaker's data to the test speaker's acoustic space. Finally, the system parameters (Gaussian means) are reestimated specifically for the test speaker using the transformed data from the selected training speakers. Experiments showed that this scheme is capable of providing an 18% relative improvement in the error rate on a large-vocabulary task with the use of as little as three sentences of adaptation data.

We present a decision-tree based procedure to quantize the feature-space of a speech recognizer, with the motivation of reducing the computation time required for evaluating gaussians in a speech recognition system. The entire feature space is quantized into non overlapping regions where each region is bounded by a number of hyperplanes. Further, each region is characterized by the occurence of only a small number of the total alphabet of allophones (sub-phonetic speech units); by identifying the region in which a test feature vector lies, only the gaussians that model the density of allophones that exist in that region need be evaluated. The quantization of the feature space is done in a heirarchical manner using a binary decision tree. Each node of the decision tree represents a region of the feature space, and is further characterized by a hyperplane (a vector v n and a scalar threshold value hn ), that subdivides the region corresponding to the current node into two non-overlapping...

Two discriminative techniques are described (and evaluated) for estimating the parameters of the Gaussians in a large vocabulary speech-recognition system. The first technique is based on using a modification of the maximum mutual information (MMI) objective function, and appears to provide no improvement over standard ML estimation. The second technique is based on a heuristic correction of the Gaussian parameters, and is seen to give a 2-5% improvement over ML estimation.

A speaker adaptation strategy is described that is based on finding a subset of speakers, from the training set, who are acoustically close to the test speaker, and using only the data from these speakers (rather than the complete training corpus) to re-estimate the system parameters. Further, a linear transformation is computed for every one of the selected training speakers to better map the training speaker's data to the test speaker's acoustic space. Finally, the system parameters (Gaussian means) are re-estimated specifically for the test speaker using the transformed data from the selected training speakers. Experiments showed that this scheme is capable of reducing the error rate by 10-15% with the use of as little as 3 sentences of adaptation data.

In this paper we discuss various experimental results using our continuous speech recognition system on the Wall Street Journal task. Experiments with different feature extraction methods, varying amounts and type of training data, and different vocabulary sizes are reported.

One area where on-line handwriting recognition technology is most critical is the domain of small portable platforms. Because such platforms have limited resources, it is not presently practical to consider a continuous parameterization for the hidden Markov models used in the recognition. On the other hand, discrete parameter techniques such as used in speech recognition are difficult to apply, because there is no well-understood handwriting equivalent to phonological rules. A possible solution is to extract this information directly from the data, by constructing an alphabet of sub-character, elementary handwriting units. The performance of this method is illustrated on a discrete handwriting recognition task with an alphabet of 81 characters.

Speaker adaptation typically involves customizing some existing (reference) models in order to account for the characteristics of a new speaker. This work considers the slightly different paradigm of customizing some reference data for the purpose of populating the new speaker's space, and then using the resulting (augmented) data to derive the customized models. The data augmentation technique is based on the metamorphic algorithm first proposed in Bellegarda et al. [1992], assuming that a relatively modest amount of data (100 sentences) is available from each new speaker. This contraint requires that reference speakers be selected with some care. The performance of this method is illustrated on a portion of the Wall Street Journal task.

In this paper we describe the method we use to derive acoustic features that reflect some of the dynamics of frame-based parameter vectors. Models for such observations must be context dependent. Such models were outlined in an earlier paper. Here we describe a method for using these models in a recognition system. The method is more robust than using continuous parameter models in recognition. At the same time it does not suffer from the possible information loss in vector quantization based systems.>

The performance of a large alphabet handwriting recognition system based on a probabilistic framework is critically tied to the quality of the prototype distributions that are established in the relevant feature space. To better account for handwriting variability, we describe a supervised strategy for the construction of prototype distributions which are more robust to allograph deformations. The idea is to incorporate supervision to relate the allographic models to their manifestations in the feature space. This makes for a better utilization of the available training data, while at the same time allowing for a short design time turn around. The performance of this method is illustrated on a discrete handwriting recognition task with an alphabet of 81 characters.>

Conventional speech recognition systems such as the IBM Tangora tend to be adversely influenced by external factors such as background noise and microphone characteristics. This paper discusses some adaptation techniques to counteract such influences. We also consider ways to reduce computation so that the methods may be implemented for real time Tangora operation. The adaptation strategy utilized two sets of VQ codebooks derived from the training data, one representing the ambience and the other characterizing the speech domain. A speech versus ambience decision was made by examining several factors, such as a comparison of the overall energy in a frame with a percentile point of a running energy histogram. We include results to demonstrate the effectiveness of our approach.>

This paper presents a channel-bank based phone detection algorithm, that can be used in greatly cut down the search space in the process of mapping a set of acoustic features to a phone sequence. The algorithm involves a low cost preprocessing, and relies on using local information (in time) about the acoustic features, to predict, whether or not a phone can exist over a certain time interval; hence, it may be thought of as producing a short-list of possible phones at all times, and it is only necessary to search in this short list, rather than in the entire phone alphabet, for the correct phone. The technique was applied in the acoustic fast match of the IBM speech recognition system, and resulted in a significant reduction in the execution time of the fast match.>

The automatic recognition of online handwriting is considered from an information theoretic viewpoint. Emphasis is placed on the recognition of unconstrained handwriting, a general combination of cursively written word fragments and discretely written characters. Existing recognition algorithms, such as elastic matching, are severely challenged by the variability inherent in unconstrained handwriting. This motivates the development of a probabilistic framework suitable for the derivation of a fast statistical mixture algorithm. This algorithm exhibits about the same degree of complexity as elastic matching, while being more flexible and potentially more robust. The approach relies on a novel front-end processor that, unlike conventional character or stroke-based processing, articulates around a small elementary unit of handwriting called a frame. The algorithm is based on (1) producing feature vectors representing each frame in one (or several) feature spaces, (2) Gaussian K-means clustering in these spaces, and (3) mixture modeling, taking into account the contributions of all relevant clusters in each space. The approach is illustrated by a simple task involving an 81-character alphabet. Both writer-dependent and writer-independent recognition results are found to be competitive with their elastic matching counterparts.>

Large vocabulary speaker-dependent speech recognition systems adjust to the acoustic peculiarities of each new speaker based on some enrolment data provided by this speaker. As the amount of data required increases with the sophistication of the underlying acoustic models, the enrolment may get lengthy. To streamline it, it is therefore desirable to make use of previously acquired speech data. The authors describe a data augmentation strategy based on a piecewise linear mapping between the feature space of a new speaker and that of a reference speaker. This speaker-normalizing mapping is used to transform the previously acquired data of the reference speaker onto the space of the new speaker. The performance of the resulting procedure, dubbed the metamorphic algorithm, is illustrated on an isolated utterance speech recognition task with a vocabulary of 20000 words. Results show that the metamorphic algorithm can substantially reduce the word error rate when only a limited amount of enrolment data is available. Alternatively, it leads to a level of performance comparable to that obtained when a much greater amount of enrolment data is required from the new speaker. In addition, it can also be used for tracking spectral evolution over time, thus providing a possible means for robust speaker self-adaptation.>

A statistical technique for vector quantizer (VQ) prototype adaptation, based on tied-mixture continuous-parameter HMM's, is derived and evaluated on the basis of experimental evidence. Performance on difficult adaptation tasks indicates that VQ-prototype adaptation via tied-mixture HMM's constitutes a useful mechanism for speaker adaptation, particularly when there are substantial channel differences or when there is a large mismatch between reference and target speaker characteristics.

A new class of hidden Markov models is proposed for the acoustic representation of words in an automatic speech recognition system. The models, built from combinations of acoustically based sub-word units called fenones, are derived automatically from one or more sample utterances of a word. Because they are more flexible than previously reported fenone-based word models, they lead to an improved capability of modeling variations in pronunciation. They are therefore particularly useful in the recognition of continuous speech. In addition, their construction is relatively simple, because it can be done using the well-known forward-backward algorithm for parameter estimation of hidden Markov models. Appropriate reestimation formulas are derived for this purpose. Experimental results obtained on a 5000-word vocabulary natural language continuous speech recognition task are presented to illustrate the enhanced power of discrimination of the new models.>

In a large vocabulary real-time speech recognition system, there is a need for a fast method for selecting a list of candidate words from the vocabulary that match well with a given acoustic input. The authors describe a highly accurate fast acoustic match for continuous speech recognition. The algorithm uses allophonic models and efficient search techniques to select a set of candidate words. The allophonic models are derived by constructing decision trees that query the context in which each phone occurs to arrive at an allophone in a given context. The models for all the words in the vocabulary are arranged in a tree structure and efficient tree search algorithms are used to select a list of candidate words using these models. Using this method, the authors are able to obtain over 99% accuracy in the fast match for a continuous speech recognition task which has a vocabulary of 5000 words.>

The authors report on a series of experiments in which the hidden Markov model baseforms and the language model probabilities were updated from spontaneously dictated speech captured during recognition sessions with the IBM Tangora system. The basic technique for baseform modification consisted of constructing new fenonic baseforms for all recognized words. To modify the language model probabilities, a simplified version of a cache language model was implemented. The word error rate across six talkers was 3.7%. Baseform adaptation reduced the average error rate to 3.5%, and using the cache language model reduced the error rate to 3.2%. Combining both techniques further reduced the error rate to 3.1%-a respectable improvement over the original error rate, especially given that the system was speaker-trained prior to adaptation.>

In a large vocabulary speech recognition system, it is desirable to make use of previously acquired speech data when encountering new speakers. The authors describe an adaptation strategy based on a piecewise linear mapping between the feature space of a new speaker and that of a reference speaker. This speaker-normalizing mapping is used to transform the previously acquired parameters of the reference speaker onto the space of the new speaker. This results in a robust speaker adaptation procedure which allows for a drastic reduction in the amount of training data required from the new speaker. The performance of this method is illustrated on an isolated utterance speech recognition task with a vocabulary of 20000 words.>

A technique for constructing hidden Markov models for the acoustic representation of words is described. The models, built from combinations of acoustically based subword units called fenones, are derived automatically from one or more sample utterances of words. They are more flexible than previously reported fenone-based word models and lead to an improved capability of modeling variations in pronunciation. In addition, their construction is simplified, because it can be done using the well-known forward-backward algorithm for the parameter estimation of hidden Markov models. Experimental results obtained on a 5000-word vocabulary continuous speech recognition task are presented to illustrate some of the benefits associated with the new models. Multonic baseforms resulted in a reduction of 16% in the average error rate obtained for ten speakers.>

The authors describe a series of experiments in which the phonetic baseform is deduced automatically for new words by utilizing actual utterances of the new word in conjunction with a set of automatically derived spelling-to-sound rules. Recognition performance was evaluated on new words spoken by two different speakers when the phonetic baseforms were extracted via the above approach. The error rates on these new words were found to be comparable to or better than when the phonetic baseforms were derived by hand, thus validating the basic approach.>

The authors present an automatic method for modeling phonological variation using decision trees. For each phone they construct a decision tree that specifies the acoustic realization of the phone as a function of the context in which it appears. Several-thousand sentences from a natural language corpus spoken by several speakers are used to construct these decision trees. Experimental results on a 5000-word vocabulary natural language speech recognition task are presented.>

The authors seek a fast algorithm for finding the best question to ask (i.e., best split of predictor values) about a predictor variable when predicting membership in more than two categories. They give a fast iterative algorithm for finding a suboptimal question in the N category problem by exploiting a fast algorithm for finding the optimal question in the two-category problem. The algorithm has been used in a number of speech recognition applications.>

The well-known Baum-Eagon inequality (1967) provides an effective iterative scheme for finding a local maximum for homogeneous polynomials with positive coefficients over a domain of probability values. However, in many applications the goal is to maximize a general rational function. In view of this, the Baum-Eagon inequality is extended to rational functions. Some of the applications of this inequality to statistical estimation problems are briefly described.>

Method for constructing groups of acoustically similar words in a large vocabulary speech recognition system are studied. The study is based on the idea that given a dictionary of words one should be able to take each word and construct a reasonably short list of words from the vocabulary that sound similar to the chosen word. Groups of acoustically similar words are constructed using the hidden Markov models for the words. They are then used in a fast match scheme in an isolated utterance recognition system. The experiments show that a short list of words can be constructed during recognition in a very quick fashion using this scheme and that the list will contain the correct word with a high probability.>

A description is presented of the authors' current research on automatic speech recognition of continuously read sentences from a naturally-occurring corpus: office correspondence. The recognition system combines features from their current isolated-word recognition system and from their previously developed continuous-speech recognition system. It consists of an acoustic processor, an acoustic channel model, a language model, and a linguistic decoder. Some new features in the recognizer relative to the isolated-word speech recognition system include the use of a fast match to prune rapidly to a manageable number the candidates considered by the detailed match, multiple pronunciations of all function words, and modeling of interphone coarticulatory behavior. The authors recorded training and test data from a set of ten male talkers. The perplexity of the test sentences was found to be 93; none of sentences was part of the data used to generate the language model. Preliminary (speaker-dependent) recognition results on these talkers yielded an average word error rate of 11.0%.>

A rapid method is presented for identifying a short list of candidate words that match well with some acoustic input to serve as a fast matching stage in a large-vocabulary speech recognition system that uses hidden Markov models and maximum a posteriori decoding. Given hidden Markov models for all the words in the vocabulary the authors derive a class of algorithms that are faster than a detailed likelihood computation using these models by constructing an estimator of the likelihood. Using such an estimator they produce a list of candidate words that match well with the given acoustic input which has the property that it is guaranteed to contain the correct word in all the cases where a detailed likelihood computation would assign the maximum likelihood to that word.>

The acoustic modeling problem in automatic speech recognition is estimated with the specific goal of unifying discrete and continuous parameter approaches. The authors consider a class of very general hidden Markov models which can accommodate sequences of information-bearing acoustic feature vectors lying either in a discrete or in a continuous space. More generally, the new class allows one to represent the prototypes in an assumption-limited, yet convenient, way, as (tied) mixtures of simple multivariate densities. Speech recognition experiments, reported for a large (5000-word) vocabulary office correspondence task, demonstrate some of the benefits associated with this technique.>

The well-known Baum-Eagon (1967) inequality provides an effective iterative scheme for finding a local maximum for homogeneous polynomials with positive coefficients over a domain of probability values. However, in a large class of statistical problems, such as those arising in speech recognition based on hidden Markov models, it was found that estimation of parameters via some other criteria that use conditional likelihood, mutual information, or the recently introduced H-criteria can give better results than maximum-likelihood estimation. These problems require finding maxima for rational functions over domains of probability values, and an analog of the Baum-Eagon inequality for rational functions makes it possible to use an E-M (expectation-maximization) algorithm for maximizing these functions. The authors describe this extension.>

The authors generalize the maximum likelihood and related optimization criteria for training and decoding with a speech recognizer. The generalizations are constructed by considering weighted linear combinations of the logarithms of the likelihoods of words, of acoustics, and of (word, acoustic) pairs. The utility of various patterns of weights are examined.>

A probabilistic mixture model is described for a frame (the short-term spectrum) of each to be used in speech recognition. Each component of the mixture is regarded as a prototype for the labeling phase of a hidden Markov model based speech recognition system. Since the ambient noise during recognition can differ from the ambient noise present in the training data, the model is designed for convenient updating in changing noise. Based on the observation that the energy in a frequency band is at any fixed time dominated either by signal energy or by noise energy, the authors model the energy as the larger of the separate energies of signal and noise in the band. Statistical algorithms are given for training this as a hidden variables model. The hidden variables are the prototype identities and the separate signal and noise components. A series of speech recognition experiments that successfully utilize this model is also discussed.>

A general technique termed adaptive labeling is presented for the normalization of the speech signal. In principle, adaptive labeling is applicable to any sequence of feature vectors of a given dimension. It combines the familiar labeling process executed by a vector quantizer with an adaptive renormalization transformation of the feature vectors proposed here. Adaptive labeling is applied to speech recognition, where the particular interest lies in diminishing the degradation of performance that occurs as a result of changes in the signal characteristics following changes in ambient noise and other recording environment conditions or in response to a change in the characteristics of the talker. Results are presented for a series of experiments using soft and loud noises as well as environments in which microphone-to-speaker distances were allowed to vary. A 5000-word vocabulary with isolated word input was used.>

The Speech Recognition Group at IBM Research in Yorktown Heights has developed a real-time, isolated-utterance speech recognizer for natural language based on the IBM Personal Computer AT and IBM Signal Processors. The system has recently been enhanced by expanding the vocabulary from 5,000 words to 20,000 words and by the addition of a speech workstation to support usability studies on document creation by voice. The system supports spelling and interactive personalization to augment the vocabularies. This paper describes the implementation, user interface, and comparative performance of the recognizer.

Statistical models (prototypes) for the multivariate probability distribution of vectors (frames) of speech parameters may be utilized in various ways. If the stream of vectors is passed directly to the decoder of a continuous parameter speech recognizer then the prototypes are used by the decoder; if the recognizer has a time-synchronous labeling acoustic processor then they are used for vector quantization (labeling) and the resulting label stream is passed to the decoder; other uses are possible as well. We present a method for constructing such prototypes. This method was chosen as a compromise between describing a prototype in an assumption free way as a nonparametric density and describing it in a convenient way as a simple multivariate Gaussian density. We describe speech recognition experiments where our prototypes were trained by iteratively interleaving steps of a K-MEANS type algorithm for clustering and steps of an EM algorithm for reestimation. We present results (using a labeling acoustic processor) having significantly fewer decoding errors than our previous methods do.

The Speech Recognition Group at IBM Research in Yorktown Heights has designed a real-time, isolated-utterance speech recognizer for natural language with a 5,000-word vocabulary based on the IBM Personal Computer (PC) AT model and two IBM Signal Processors realized in VLSI technology. The enrollment period for a new user is approximately 20 minutes. The basic vocabulary is chosen from the most common words in several collections of documents such as office memoranda and business letters. The system supports spelling and interactive personalization to augment this vocabulary. Signal processing, vector quantization, and acoustic matching algorithms are programmed on the IBM Signal Processors which fit into the PC AT chassis. The PC AT controls the Processors and implements the decoder stack search and the language model, as well as the application-specific interface. The modular architecture of the design is expandable to a 20,000-word vocabulary system by the addition of two more IBM Signal Processors housed in a PC Expansion Unit.

Selection of acoustic prototypes is an important aspect of speech recognition systems. In this paper a new algorithm for obtaining a set of acoustic prototypes is described. Associated with each prototype is a distortion measure whose full quadratic distance form is optimized to achieve a local minimum for the average distortion. Using a K-means clustering strategy, it is shown that in each iteration the minimum is achieved when the eigenvectors of the weighting matrices of the qudratic distances are identical to those of the sample covariance matrices of their corresponding clusters. Under certain eigenvalue constraints, closed form solutions for finding the eigenvalues are provided. Furtheremore, it is shown that clustering schemes that assume a multivariate Gaussian mixture density for the data can be solved using the new technique. Thereby, a new derivation for the maximum likelihood estimate of their associated covariance matrices is presented. Finally, recognition results obtained by incorporating the new algorithm in the IBM Speech Recognition system are presented.

Recognition results on sentences from a 5000-word vocabulary drawn from office correspondence are presented. The sentences were read with pauses between the words. The vocabulary comprises the 5000 most frequently occurring words in a data-base of 14,000 office memoranda and letters, and has a perplexity of 90, measured from a trigram language model. Experiments were carried out with 6 speakers (4 male, 2 female) in an office environment using a close-talking microphone. The recognition system was automatically trained to each speaker by having the speaker read 100 typical sentences from the office correspondence data-base. Recognition was carried out for each speaker on 20 test sentences, consisting of 299 words. The recognition rate (% words correct) averaged across the 6 speakers was 94.5%.