We describe technology and experience with an experimental personal information manager, which interacts with the user primarily but not exclusively through speech recognition and synthesis. This device, which controls a client PDA, is known as the personal speech assistant (PSA). The PSA contains complete speech recognition, speech synthesis and dialog management systems. Packaged in a hand-sized enclosure, of size and physical design to mate with the popular Palm III personal digital assistant, the PSA includes its own battery, microphone, speaker, audio input and output amplifiers, processor and memory. The PSA supports speaker-independent English speech recognition using a 500-word vocabulary, and English speech synthesis on an arbitrary vocabulary. We survey the technical issues we encountered in building the hardware and software for this device, and the solutions we implemented, including audio system design, power and space budget, speech recognition in adverse acoustic environments with constrained processing resources, dialog management, appealing applications, and overall system architecture.

One difficult problem we are often faced with in clustering analysis is how to choose the number of clusters. We propose to choose the number of clusters by optimizing the Bayesian information criterion (BIC), a model selection criterion in the statistics literature. We develop a termination criterion for the hierarchical clustering methods which optimizes the BIC criterion in a greedy fashion. The resulting algorithms are fully automatic. Our experiments on Gaussian mixture modeling and speaker clustering demonstrate that the BIC criterion is able to choose the number of clusters according to the intrinsic complexity present in the data.

This paper describes extensions and improvements to IBM's large vocabulary continuous speech recognition (LVCSR) system for transcription of broadcast news. The recognizer uses an additional 35 hours of training data over the one used in the 1996 Hub4 evaluation. It includes a number of new features: optimal feature space for acoustic modeling (in training and/or testing), filler-word modeling, Bayesian information criterion (BIC) based segment clustering, an improved implementation of iterative MLLR and 4-gram language models. Results using the 1996 DARPA Hub4 evaluation data set are presented.

In this paper, we describe a new compression algorithm for encoding acoustic features used in typical speech recognition systems. The proposed algorithm uses a combination of simple techniques, such as linear prediction and multi-stage vector quantization, and the current version of the algorithm encodes the acoustic features at a fixed rate of 4.0 kbit/s. The compression algorithm can be used very effectively for speech recognition in network environments, such as those employing a client-server model, or to reduce storage in general speech recognition applications. The algorithm has also been tuned for practical implementations, so that the computational complexity and memory requirements are modest. We have successfully tested the compression algorithm against many test sets from several different languages, and the algorithm performed very well, with no significant change in the recognition accuracy due to compression.

In this paper, we propose a new bootstrap technique to build domain-dependent language models. We assume that a seed corpus consisting of a small amount of data relevant to the new domain is available, which is used to build a reference language model. We also assume the availability of an external corpus, consisting of a large amount of data from various sources, which need not be directly relevant to the domain of interest. We use the reference language model and a suitable metric, such as the perplexity measure, to select sentences from the external corpus that are relevant to the domain. Once we have a sufficient number of new sentences, we can rebuild the reference language model. We then continue to select additional sentences from the external corpus, and this process continues to iterate until some satisfactory termination point is achieved. We also describe several methods to further enhance the bootstrap technique, such as combining it with mixture modeling and class-based modeling. The performance of the proposed approach was evaluated through a set of experiments, and the results are discussed. Analysis of the convergence properties of the approach and the conditions that need to be satisfied by the external corpus and the seed corpus are highlighted, but detailed work on these issues is deferred for the future.

This paper describes some of the main problems and issues specific to the transcription of broadcast news and describes some of the methods for solving them that have been incorporated into the IBM Large Vocabulary Continuous Speech Recognition System.

Two discriminative techniques are described (and evaluated) for estimating the parameters of the Gaussians in a large vocabulary speech-recognition system. The first technique is based on using a modification of the maximum mutual information (MMI) objective function, and appears to provide no improvement over standard ML estimation. The second technique is based on a heuristic correction of the Gaussian parameters, and is seen to give a 2-5% improvement over ML estimation.

In this paper we discuss various experimental results using our continuous speech recognition system on the Wall Street Journal task. Experiments with different feature extraction methods, varying amounts and type of training data, and different vocabulary sizes are reported.

We describe a tree search strategy, called the Envelope Search, which is a time-asynchronous search scheme that combines aspects of the A* heuristic search algorithm with those of the time-synchronous Viterbi search algorithm. This search technique is used in the large-vocabulary continuous speech recognition system developed at the IBM Research Center.

In this paper we describe the method we use to derive acoustic features that reflect some of the dynamics of frame-based parameter vectors. Models for such observations must be context dependent. Such models were outlined in an earlier paper. Here we describe a method for using these models in a recognition system. The method is more robust than using continuous parameter models in recognition. At the same time it does not suffer from the possible information loss in vector quantization based systems.>

This paper presents a channel-bank based phone detection algorithm, that can be used in greatly cut down the search space in the process of mapping a set of acoustic features to a phone sequence. The algorithm involves a low cost preprocessing, and relies on using local information (in time) about the acoustic features, to predict, whether or not a phone can exist over a certain time interval; hence, it may be thought of as producing a short-list of possible phones at all times, and it is only necessary to search in this short list, rather than in the entire phone alphabet, for the correct phone. The technique was applied in the acoustic fast match of the IBM speech recognition system, and resulted in a significant reduction in the execution time of the fast match.>

A new class of hidden Markov models is proposed for the acoustic representation of words in an automatic speech recognition system. The models, built from combinations of acoustically based sub-word units called fenones, are derived automatically from one or more sample utterances of a word. Because they are more flexible than previously reported fenone-based word models, they lead to an improved capability of modeling variations in pronunciation. They are therefore particularly useful in the recognition of continuous speech. In addition, their construction is relatively simple, because it can be done using the well-known forward-backward algorithm for parameter estimation of hidden Markov models. Appropriate reestimation formulas are derived for this purpose. Experimental results obtained on a 5000-word vocabulary natural language continuous speech recognition task are presented to illustrate the enhanced power of discrimination of the new models.>

In a large vocabulary speech recognition system using hidden Markov models, calculating the likelihood of an acoustic signal segment for all the words in the vocabulary involves a large amount of computation. In order to run in real time on a modest amount of hardware, it is important that these detailed acoustic likelihood computations be performed only on words which have a reasonable probability of being the word that was spoken. The authors describe a scheme for rapidly obtaining an approximate acoustic match for all the words in the vocabulary in such a way as to ensure that the correct word is, with high probability, one of a small number of words examined in detail. Using fast search methods, they obtain a matching algorithm that is about a hundred times faster than doing a detailed acoustic likelihood computation on all the words in the IBM Office Correspondence isolated word dictation task, which has a vocabulary of 20000 words. Experimental results showing the effectiveness of such a fast match for a number of talkers are given.>

In a large vocabulary real-time speech recognition system, there is a need for a fast method for selecting a list of candidate words from the vocabulary that match well with a given acoustic input. The authors describe a highly accurate fast acoustic match for continuous speech recognition. The algorithm uses allophonic models and efficient search techniques to select a set of candidate words. The allophonic models are derived by constructing decision trees that query the context in which each phone occurs to arrive at an allophone in a given context. The models for all the words in the vocabulary are arranged in a tree structure and efficient tree search algorithms are used to select a list of candidate words using these models. Using this method, the authors are able to obtain over 99% accuracy in the fast match for a continuous speech recognition task which has a vocabulary of 5000 words.>

A technique for constructing hidden Markov models for the acoustic representation of words is described. The models, built from combinations of acoustically based subword units called fenones, are derived automatically from one or more sample utterances of words. They are more flexible than previously reported fenone-based word models and lead to an improved capability of modeling variations in pronunciation. In addition, their construction is simplified, because it can be done using the well-known forward-backward algorithm for the parameter estimation of hidden Markov models. Experimental results obtained on a 5000-word vocabulary continuous speech recognition task are presented to illustrate some of the benefits associated with the new models. Multonic baseforms resulted in a reduction of 16% in the average error rate obtained for ten speakers.>

The authors present an automatic method for modeling phonological variation using decision trees. For each phone they construct a decision tree that specifies the acoustic realization of the phone as a function of the context in which it appears. Several-thousand sentences from a natural language corpus spoken by several speakers are used to construct these decision trees. Experimental results on a 5000-word vocabulary natural language speech recognition task are presented.>

The well-known Baum-Eagon inequality (1967) provides an effective iterative scheme for finding a local maximum for homogeneous polynomials with positive coefficients over a domain of probability values. However, in many applications the goal is to maximize a general rational function. In view of this, the Baum-Eagon inequality is extended to rational functions. Some of the applications of this inequality to statistical estimation problems are briefly described.>

Method for constructing groups of acoustically similar words in a large vocabulary speech recognition system are studied. The study is based on the idea that given a dictionary of words one should be able to take each word and construct a reasonably short list of words from the vocabulary that sound similar to the chosen word. Groups of acoustically similar words are constructed using the hidden Markov models for the words. They are then used in a fast match scheme in an isolated utterance recognition system. The experiments show that a short list of words can be constructed during recognition in a very quick fashion using this scheme and that the list will contain the correct word with a high probability.>

A description is presented of the authors' current research on automatic speech recognition of continuously read sentences from a naturally-occurring corpus: office correspondence. The recognition system combines features from their current isolated-word recognition system and from their previously developed continuous-speech recognition system. It consists of an acoustic processor, an acoustic channel model, a language model, and a linguistic decoder. Some new features in the recognizer relative to the isolated-word speech recognition system include the use of a fast match to prune rapidly to a manageable number the candidates considered by the detailed match, multiple pronunciations of all function words, and modeling of interphone coarticulatory behavior. The authors recorded training and test data from a set of ten male talkers. The perplexity of the test sentences was found to be 93; none of sentences was part of the data used to generate the language model. Preliminary (speaker-dependent) recognition results on these talkers yielded an average word error rate of 11.0%.>

A rapid method is presented for identifying a short list of candidate words that match well with some acoustic input to serve as a fast matching stage in a large-vocabulary speech recognition system that uses hidden Markov models and maximum a posteriori decoding. Given hidden Markov models for all the words in the vocabulary the authors derive a class of algorithms that are faster than a detailed likelihood computation using these models by constructing an estimator of the likelihood. Using such an estimator they produce a list of candidate words that match well with the given acoustic input which has the property that it is guaranteed to contain the correct word in all the cases where a detailed likelihood computation would assign the maximum likelihood to that word.>

The well-known Baum-Eagon (1967) inequality provides an effective iterative scheme for finding a local maximum for homogeneous polynomials with positive coefficients over a domain of probability values. However, in a large class of statistical problems, such as those arising in speech recognition based on hidden Markov models, it was found that estimation of parameters via some other criteria that use conditional likelihood, mutual information, or the recently introduced H-criteria can give better results than maximum-likelihood estimation. These problems require finding maxima for rational functions over domains of probability values, and an analog of the Baum-Eagon inequality for rational functions makes it possible to use an E-M (expectation-maximization) algorithm for maximizing these functions. The authors describe this extension.>

The authors generalize the maximum likelihood and related optimization criteria for training and decoding with a speech recognizer. The generalizations are constructed by considering weighted linear combinations of the logarithms of the likelihoods of words, of acoustics, and of (word, acoustic) pairs. The utility of various patterns of weights are examined.>