This paper describes a set of 58 Matlab®-based speech processing exercises designed to give students and instructors hands-on experience with digital speech processing basics, fundamentals, representations, algorithms and applications. This result is achieved by providing working Matlab code using a LITE graphical user interface (GUI) for ease of use and understanding of the code. For each Matlab exercise we provide a brief description of the code and GUI structure, an explanation of key technical aspects of the solution, a description of the organization of the Lite Graphical User Interface, a scripted run of the software with an extensive set of speech files for processing, some results figures generated by the provided code, and a set of technical issues for further experimentation with the code. Our goal with these exercises is to provide a set of speech processing tools for the instructor and an extended set of speech processing exercises (denoted as “Issues for Further Experimentation”) that students can use to develop their understanding of the technical content of each speech processing exercise. This paper describes the resulting set of exercises and shows how they can be used to enhance the teaching and learning of digital speech processing.

A new loss function has been introduced for Minimum Classification Error, that approaches optimal Bayes’ risk and also gives an improvement in performance over standard MCE systems when evaluated on the Aurora connected digits database.

A Minimum Classification Error (MCE) based recognition system that also estimates a global feature transformation matrix has been implemented. Unlike earlier studies, we make the explicit assumption that the covariance matrix of the Gaussian mixtures is diagonal when estimating the transformation matrix. This is necessary for mathematical consistency between the model and the transformation matrix estimates. Experimental results show a reduction of up to 50% in the word error rate as compared to Maximum Likelihood estimation.

Automatic Speech Attribute Transcription (ASAT), an ITR project sponsored under the NSF grant (IIS-04-27113), is a cross-institute effort involving Georgia Institute of Technology, The Ohio State University, University of California at Berkeley, and Rutgers University. This project approaches speech recognition from a more linguistic perspective: unlike traditional ASR systems, humans detect acoustic and auditory cues, weigh and combine them to form theories, and then process these cognitive hypotheses until linguistically and pragmatically consistent speech understanding is achieved. A major goal of the ASAT paradigm is to develop a detection-based approach to automatic speech recognition (ASR) based on attribute detection and knowledge integration. We report on progress of the ASAT project, present a sharable platform for community collaboration, and highlight areas of potential interdisciplinary ASR research. Index Terms: attributes, events, features, detection, speech recognition, speech attribute transcription, utterance verification

In this paper we discuss the design and implementation of the ASAT front end processing system, whose goal is to convert the speech waveform into a range of measurements and parameters which are then combined to form probabilistic attributes. The ASAT front end processing module utilizes a range of spectral and temporal speech parameters as input to a set of neural network classifiers to create sets of attribute probability lattices, based on either single frames or blocks of frames (segments). We test this architecture by using the 14 Sound Patterns of English (SPE) features as speech attributes. Without balancing the training data, the detection accuracies of 4 of the SPE features are above 90%, 2 features obtain between 80% and 90% detection accuracy, and 8 features have detection accuracies below 80%. With a novel method of balancing the feature training data, the performance of the neural networks improved significantly, with 6 features having detection accuracies above 90% and the remaining 8 features with detection accuracy above 80%

Earlier research has shown that the maximum spectral transition positions are related with the perceptual critical points that contain the most important information for consonant and syllable perception. This paper presents a quantitative analysis of the relation, in time, between the maximum spectral transition positions and the phone boundaries in fluent read speech. This analysis is based on the training part of the TIMIT American English database which contains both phone boundaries and labels manually-determined by a group of experts. The results of this analysis show that there is a significant correlation between the maximum spectral transition positions and the manually selected phone boundaries. This, in turn, suggests that there is an important relation between the commonly accepted phone boundaries and the perceptual critical points. Index Terms: phone boundaries, spectral transition, phonemes 1.

In spite of the effort and progress made during the last few decades, the performance of automatic speech recognition (ASR) systems still lags far behind that achieved by humans. Some researchers think that more speech data will be sufficient in order to bridge this performance gap. Others think that radical modifications to the current methods need to be made, and possible inspirations for these modifications should come from human speech perception (HSP). This paper focuses on two issues: first, it presents a comparison between HSP and ASR emphasizing some insights from HSP that could still be applied in ASR; second, it presents some ideas for extracting useful non-linguistic information from the speech signal, the so called ‘rich transcription’, which could help in selecting specialized acoustic-linguistic models that offer higher accuracy than the general models. 1.

We present a novel paradigm for human to human asymmetric collaboration. There is a need for people at geographically separate locations to seamlessly collaborate in real time as if they are physically co-located. In our system one user (novice) works in the real world and the other user (expert) works in a parallel virtual world. They are assisted in this task by an Intelligent Agent (IA) with considerable knowledge about the environment. Current tele-collaboration systems deal primarily with collaboration purely in the real or virtual worlds. The use of a combination of virtual and real worlds allows us to leverage the advantages from both the worlds.

In the future, the world of telecommunications will be vastly different than it is today. The driving force will be the seamless integration of real time communications (e.g. voice, video, music, etc.) and data into a single network, with ubiquitous access to that network anywhere, anytime, and by a wide range of devices. The only currently available ubiquitous access device to the network is the telephone, and the only ubiquitous user access technology mode is spoken voice commands and natural language dialogues with machines. In the future, new access devices and modes will augment speech in this role, but are unlikely to supplant the telephone and access by speech anytime soon. Speech technologies have progressed to the point where they are now viable for a broad range of communications services, including: compression of speech for use over wired and wireless networks; speech synthesis, recognition, and understanding for dialogue access to information, people, and messaging; and speaker verification for secure access to information and services. The paper provides brief overviews of these technologies, discusses some of the unique properties of wireless, plain old telephone service, and Internet protocol networks that make voice communication and control problematic, and describes the types of voice services available in the past and today, and those that we foresee becoming available over the next several years.

The challenge of multimedia processing is to provide services that seamlessly integrate text, sound, image, and video information and to do it in a way that preserves the ease of use and interactivity of conventional plain old telephone service (POTS) telephony. To achieve this goal, there are a number of technological problems that must be considered, including: compression and coding of multimedia signals, including algorithmic issues, standards issues, and transmission issues; synthesis and recognition of multimedia signals, including speech, images, handwriting, and text; organization, storage, and retrieval of multimedia signals, including the appropriate method and speed of delivery, resolution, and quality of service; access methods to the multimedia signal, including spoken natural language interfaces, agent interfaces, and media conversion tools; searching by text, speech, and image queries; browsing by accessing the text, by voice, or by indexed images. In each of these areas, a great deal of progress has been made in the past few years, driven in part by the relentless growth in multimedia personal computers and in part by the promise of broad-band access from the home and from wireless connections. Standards have also played a key role in driving new multimedia services, both on the POTS network and on the Internet. It is the purpose of this paper to review the status of the technology in each of the areas listed above and to illustrate current capabilities by describing several multimedia applications that have been implemented at AT&T Labs over the past several years.

Discusses coding standards for still images and motion video. We first briefly discuss standards already in use, including: Group 3 and Group 4 for bilevel fax images; JPEG for still color images; and H.261, H.263, MPEG-1, and MPEG-2 for motion video. We then cover newly emerging standards such as JBIG1 and JBIG2 for bilevel fax images, JPEG-2000 for still color images, and H.263+ and MPEG-4 for motion video. Finally, we describe some directions beyond the standards such as hybrid coding of graphics/photo images, MPEG-7 for multimedia metadata, and possible new technologies.

The challenge of multimedia processing is to seamlessly integrate text, sound, image, and video information into a single communications channel, and to do it in a way that provides high quality communications while preserving the ease-of-use and interactivity of conventional telephony. There are a number of technology drivers that are pushing the technology forward, as well as a number of technological problems that must be overcome before multimedia becomes as ubiquitous as voiceband telephony. A key issue with any practical multimedia system has to do with standards that insure connectivity between customers and a range of service providers. Multimedia processing is an area of communications that is rapidly evolving. However, a number of interesting and important multimedia communications applications have evolved over the past several years, and some of these applications are described.

The ways in which people communicate are changing rapidly. The options are many and diverse, ranging from voice calls over wireless networks, to video calls over the conventional wired network, ISDN video, FAX, e-mail, voice mail, beeper services, data services, audio teleconferencing, video teleconferencing, and so-called scribble phone service (transmission of arbitrary handwritten input). This revolution in communications is being fueled by several sources, including the availability of low-cost, low-power, computation in both DSP and RISC chips, larger and cheaper memory chips, improved algorithms for communications (e.g., modems, signaling) and signal processing, and finally the creation of world-wide standards for transmission, signal compression, and communication protocols. The broad goal of the communications revolution is to provide seamless and high-quality communications between people (or groups of people), anywhere, anytime, and at a reasonable price. Although there are many technologies that form the bases for the communications environment of the twenty-first century, one of the key technologies for making the vision a reality is voice processing. In this paper we attempt to show, by example, how voice processing has been applied to specific problems in telecommunications, and how it will grow to become an even more essential component of the communications systems of the twenty-first century.>

The authors report recent improvements to an HMM (hidden, Markov model)-based, continuous speech recognition system. These advances, which include the incorporation of interword context-dependent units and position-dependent units and an improved feature analysis, lead to a recognition system which gives a 95% word accuracy and 75% sentence accuracy for speaker independent recognition of the 1000-word, DARPA resource management task using the standard word pair grammar (with a perplexity of about 60). With the improved acoustic modeling of subword units, the overall error rate reduction was over 42% compared with the performance results reported in the baseline system. The best results obtained so far using the word pair grammar gave 95.2% average word accuracy for the three DARPA evaluation sets.>

The authors provide a detailed description of all aspects of the implementation of a large-vocabulary speaker-independent, continuous speech recognizer used as a tool for the development of recognition algorithms based on hidden Markov models (HMMs) and Viterbi decoding. The complexity of HMM recognizers is greatly increased by the introduction of detailed context-dependent units for representing interword coarticulation. A vectorized representation of the data structures involved in the decoding process, along with compilation of the connection information among temporally consecutive words and an efficient implementation of the beam search pruning, has led to a speedup of the algorithm of about one order of magnitude. A guided search can be used during a tuning phase for obtaining a speedup of more than three times. An average recognition time of about 25 s per sentence, although far from real time, allows one to perform a series of training experiments and to tune the recognition system parameters in order to obtain high word accuracy on complex recognition tasks such as the DARPA resource management task.>

It is shown how one can apply the improved acoustic modeling techniques (using a continuous density hidden Markov model framework) developed for large vocabulary speech recognition applications to the problem of connected digit recognition with no changes made to the basic modeling techniques and with no vocabulary specific information used. The improved modeling techniques adopted in this study include an improved feature analysis procedure, which incorporates higher order cepstral and log energy time derivatives, and an improved acoustic resolution procedure, which uses more Gaussian mixture components per state to characterize the acoustic variability in each state of the model. Using these techniques, string accuracies of 98.6% for unknown length strings and 99.2% for known length strings were achieved on the standard Texas Instruments connected digits database.>

Acoustic modeling method of basic speech subword units are discussed to provide high word recognition accuracy. It is shown that for a basic set of 47 context-independent phone-like units, word accuracies on the order of 86-90% can be obtained for a 1000-word vocabulary, in a speaker-independent mode, for a grammar with a perplexity of 60, on independent test sets. When the basic set of units is increased to include context-dependent units, word recognition accuracies of from 91 to 93% can be achieved on the same test sets. Based on outside results and some of the present ones, it is possible to increase word recognition accuracies by about 2-3% using known modeling techniques.>

An approach for designing a set of acoustic models for speech recognition applications which results in a minimal empirical error rate for a given decoder and training data is studied. In an evaluation of the system for an isolated word recognition task, hidden Markov models (HMMs) are used to characterize the probability density functions of the acoustic signals from the different words in the vocabulary. Decoding is performed by applying the maximum aposteriori decision rule to the acoustic models. The HMMs are estimated by minimizing a differentiable cost function, which approximates the empirical error rate function, using the steepest descent method. The HMMs designed by the minimum empirical error rate approach were used in multispeaker recognition of the English E-set words and compared to models designed by the standard maximum-likelihood estimation approach. The approach increased recognition accuracy from 68.2% to 76.2% on the training set and from 53.4% to 56.4% on an independent set of test data.>

Some relations among approaches that have been applied to estimating models for acoustic signals in speech recognition systems are examined. In particular, the modeling approaches based on maximum likelihood (ML), maximum mutual information (MMI), and minimum discrimination information (MDI) are studied. It is shown that all three approaches can be formulated uniformly as MDI modeling approaches for simultaneous estimation of the acoustic models for all words in the vocabulary and that none of the approaches requires any model correctness assumption. The three approaches differ in the effective source being modeled and in the probability distribution attributed to this source.>

The problem of how to select and construct a set of fundamental unit statistical models suitable for speech recognition is addressed. A unified framework is discussed which can be used to accomplish the goal of creating effective basic models of speech. The performances of three types of fundamental units, namely whole word, phoneme-like, and acoustic segment units, in a 1109-word vocabulary speech recognition task are compared. The authors point out the relative advantages of each type of speech unit based on the results of a series of recognition experiments.>

The authors describe an HMM (hidden Markov model) clustering procedure and discuss its application to connected-word systems and to large-vocabulary recognition based on phonelike units. It is shown that the conventional approach of maximizing likelihood is easily implemented but does not work well in practice, as it tends to give improved models of tokens for which the initial model was generally quite good, but does not improve tokens which are poorly represented by the initial model. The authors have developed a splitting procedure which initializes each new cluster (statistical model) by splitting off all tokens in the training set which were poorly represented by the current set of models. This procedure is highly efficient and gives excellent recognition performance in connected-word tasks. In particular, for speaker-independent connected-digit recognition, using two HMM-clustered models, the recognition performance is as good as or better than previous results using 4-6 models/digit obtained from template-based clustering.>

The authors present an algorithm based on hidden Markov models which can recognize a predefined set of vocabulary items spoken in the context of fluent speech. They show that for a vocabulary of five words, it is possible to correctly recognize 87.1% of keywords when they occur in fluent speech and are spoken over a long-distance telephone network. While this task is significantly easier than what is normally associated with keyword spotting in continuous speech, it does address an important problem that must be solved for successful deployment of speech-recognition technology.>

This tutorial provides an overview of the basic theory of hidden Markov models (HMMs) as originated by L.E. Baum and T. Petrie (1966) and gives practical details on methods of implementation of the theory along with a description of selected applications of the theory to distinct problems in speech recognition. Results from a number of original sources are combined to provide a single source of acquiring the background required to pursue further this area of research. The author first reviews the theory of discrete Markov chains and shows how the concept of hidden states, where the observation is a probabilistic function of the state, can be used effectively. The theory is illustrated with two simple examples, namely coin-tossing, and the classic balls-in-urns system. Three fundamental problems of HMMs are noted and several practical techniques for solving these problems are given. The various types of HMMs that have been studied, including ergodic as well as left-right models, are described.>

An iterative approach for minimum-discrimination-information (MDI) hidden Markov modeling of information sources is proposed. The approach is developed for sources characterized by a given set of partial covariance matrices and for hidden Markov models (HMMs) with Gaussian autoregressive output probability distributions (PDs). The approach aims at estimating the HMM which yields the MDI with respect to all sources that could have produced the given set of partial covariance matrices. Each iteration of the MDI algorithm generates a new HMM as follows. First, a PD for the source is estimated by minimizing the discrimination information measure with respect to the old model over all PDs which satisfy the given set of partial covariance matrices. Then a new model that decreases the discrimination information measure between the estimated PD of the source and the PD of the old model is developed. The problem of estimating the PD of the source is formulated as a standard constrained minimization problem in the Euclidean space. The estimation of a new model given the PD of the source is done by a procedure that generalizes the Baum algorithm. The MDI approach is shown to be a descent algorithm for the discrimination information measure, and its local convergence is proved.>

The authors examine the relations between maximum likelihood (ML), maximum mutual information (MMI), and minimum discrimination information (MDI) modeling approaches, which have been applied to estimating acoustic word models in speech recognition systems. The show that all three approaches can be uniformly formulated as MDI modeling approaches for estimating the acoustic models for all words simultaneously. The three approaches differ in either the probability distribution (PD) attributed to the source being modeled or in the model effectively being used. None of the approaches, however, assumes model correctness, i.e., that the source has the PD of the model. A new modeling approach is proposed, which, in contrast with the other approaches considered, directly aims at the minimization of the probability of error.>

A network-based, frame-synchronous, level building (FSLB) algorithm is described for recognizing continuous speech as a connected sequence of words. Previous frame synchronous approaches to recognition have included the one-pass approach, the one-stage approach, and an unpublished multilevel approach. The proposed algorithm, which has all the features of these earlier methods, as well as several new features, and which is implemented in a network-based approach, is a numerically exact, frame synchronous, implementation of the conventional level building (LB) algorithm. As with some of the earlier methods, the proposed algorithm is highly regular and modularized for distributed computation among several spectral purpose processors. New features of the algorithm include the capability of determining the best alternative recognition strings (e.g. second and third best strings), at every level, even for very complicated grammar networks, and the capability of efficiently incorporating several word and state duration scoring techniques directly in the forward search thereby eliminating the need for a postprocessor as required in the direct LB implementation. Word transition rules (e.g. a language model) can also be easily incorporated into the proposed algorithm.>

Past research has shown that a connected digit recognition system, based on either word templates or word hidden Markov models (HMM), could effectively be trained using a segmental k-means training procedure. In these studies, a set of randomly generated digit strings of variable length was used to train the recognizer. However, problems were encountered when this training procedure was extended to systems with medium to large vocabularies. For a training set to be effective, it should represent each vocabulary word and its acoustic variability within the context of all valid input strings defined by a task dependent grammar. A procedure to generate a training set of sentences with these properties is proposed. Using this procedure, a training set of sentences was generated for a connected word recognition system simulating an airline flight reservation task. Several speaker dependent automatic speech recognition (ASR) experiments were performed to assess the effectiveness of the training set generated using the new procedure. The results of these experiments showed that the string accuracy was about 98% when tested on independent sets of test sentences for the three talkers.>

Algorithms for connected-word recognition based on whole-word reference patterns have become increasingly sophisticated and have been shown capable of achieving high recognition performance for small or syntax-constrained moderate-size vocabularies in a speaker-trained mode. An enhanced analysis feature set consisting of both instantaneous and transitional spectral information is used and the hidden-Markov-model-based connected digit recognizer is tested in speaker-trained, multispeaker, and speaker-independent modes. The performance achieved was 0.35, 1.65 and 1.75% string error rates, respectively, for known length strings and 0.78, 2.85 and 2.94% string error rates, respectively, for unknown length strings.>

A new iterative approach for hidden Markov modeling of information sources which aims at minimizing the discrimination information (or the cross-entropy) between the source and the model is proposed. This approach does not require the commonly used assumption that the source to be modeled is a hidden Markov process. The algorithm is started from the model estimated by the traditional maximum likelihood (ML) approach and alternatively decreases the discrimination information over all probability distributions of the source which agree with the given measurements and all hidden Markov models. The proposed procedure generalizes the Baum algorithm for ML hidden Markov modeling. The procedure is shown to be a descent algorithm for the discrimination information measure and its local convergence is proved.

We investigate the performance of a recent algorithm for linear predictive (LP) modeling of speech signals, which have been degraded by uncorrelated additive noise, as a front-end processor in a speech recognition system. The system is speaker dependent, and recognizes isolated words, based on dynamic time warping principles. The LP model for the clean speech is estimated through appropriate composite modeling of the noisy speech. This is done by minimizing the Itakura-Saito distortion measure between the sample spectrum of the noisy speech and the power spectral density of the composite model. This approach results in a "filtering-modeling" scheme in which the filter for the noisy speech, and the LP model for the clean speech, are alternatively optimized. The proposed system was tested using the 26 word English alphabet, the ten English digits, and the three command words, "stop," "error," and "repeat," which were contaminated by additive white noise at 5-20 dB signal to noise ratios (SNR's). By replacing the standard LP analysis with the proposed algorithm, during training on the clean speech and testing on the noisy speech, we achieve an improvement in recognition accuracy equivalent to an increase in input SNR of approximately 10 dB.

Traditional approaches to the problem of noise suppression or signal restoration have been almost entirely based upon the methodology and theory of signal estimation. In this paper, we treat signal restoration as a problem in signal detection. Instead of estimating the characteristics of the signal and/or the noise, we establish a correspondence between the clean and the noisy signal through spectral mapping. In the procedure, we collect separate samples of both the clean signal and the noise. When the noise is additive, the (simulated) noisy signal is obtained by adding the noise to the clean signal. The sequence of short time spectra of the clean signal and that of the noisy signal form a one-to-one correspondence. The noisy spectral sequence is then used as a detection reference, to which the short time spectrum of an unknown noisy observation is compared, resulting in a detected occurrence of a particular group of spectra in the noisy sequence. Through the (inverse) mapping, the clean spectra that correspond to the detected noisy spectra are selected and processed to produce the restored spectrum. One important notion of the approach is that it is not limited to the usual least squares or minimum mean square framework. Our preliminary results show that when the mapping (detection) is based upon the likelihood ratio distortion measure, an SNR improvement of approximately 10 dB is obtainable for a 14 dB SNR noisy signal. Under the same condition, an improvement of approximately 8.5 dB can be obtained using a truncated cepstral distance measure.

In this paper we discuss a system for automatically recognizing fluently spoken digit strings based on whole word reference units. The system that we will describe can use either hidden Markov model (HMM) technology or template-based technology. The training procedure derives the digit reference patterns (either templates or statistical models) from connected digit strings. To evaluate the performance of the overall connected digit recognizer, a set of 50 people (25 men, 25 women), from the non-technical local population, was each asked to record 1200 random connected digit strings over local dialed-up telephone lines. Both a speaker trained and a multispeaker training set was created, and a full performance evaluation was made. Results show that the average string accuracy for unknown and known length strings, in the speaker trained mode, was 98% and 99% respectively; in the multi-speaker mode the average string accuracies were 94% and 96.6% respectively.

An approach to automatic speech recognition is described which attempts to link together ideas from pattern recognition such as dynamic time warping and hidden Markov modeling, with ideas from linguistically motivated approaches. In this approach, the basic sub-word units are defined acoustically, but not necessarily phonetically. An algorithm was developed which automatically decomposed speech into multiple sub-word segments, based solely upon strict acoustic criteria, without any reference to linguistic content. By repeating this procedure on a large corpus of speech data we obtained an extensive pool of unlabeled sub-word speech segments. Then using well defined clustering techniques, a small set of representative acoustic sub-word units (e.g. an inventory of units) was created. This process is fast, easy to use, and required no human intervention. The interpretation of these sub-word units, in a linguistic sense, in the context of word decoding is an important issue which must be addressed for them to be useful in a large vocabulary system. We have not yet addressed this issue; instead a couple of simple experiments were performed to determine if these acoustic sub-word units had any potential value for speech recognition. For these experiments we used a connected digits database from a single female talker. A 25 sub-word unit codebook of acoustic segments was created from about 1600 segments drawn from 100 connected digit strings. A simple isolated digit recognition system, designed using the statistics of the codewords in the acoustic sub-word unit codebook had a recognition accuracy of 100%. In another experiment a connected digit recognition system was created with representative digit templates created by concatenating the sub-word units in an appropriate manner. The system had a string recognition accuracy of 96%.

In this paper a signal modeling technique based upon finite mixture autoregressive probabilistic functions of Markov chains is developed and applied to the problem of speech recognition, particularly speaker-independent recognition of isolated digits. Two types of mixture probability densities are investigated: finite mixtures of Gaussian autoregressive densities (GAM) and nearest-neighbor partitioned finite mixtures of Gaussian autoregressive densities (PGAM). In the former (GAM), the observation density in each Markov state is simply a (stochastically constrained) weighted sum of Gaussian autoregressive densities, while in the latter (PGAM) it involves nearest-neighbor decoding which, in effect, defines a set of partitions on the observation space. In this paper we discuss the signal modeling methodology and give experimental results on speaker independent recognition of isolated digits.

In this paper, we extend the interpretation of distortion measures, based upon the observation that measurements of speech spectral envelopes (as normally obtained from analysis procedures) are prone to statistical variations due to window position fluctuations, excitation interference, measurement noise, etc. and may possess spurious characteristics because of analysis model constraints. We have found that these undesirable spectral measurement variations can be controlled (i.e. reduced in the level of variation) through proper cepstral processing and that a statistical model can be established to predict the variances of the cepstral coefficient measurements. The findings lead to the use of a bandpass "liftering" process aimed at reducing the variability of the statistical components of spectral measurements. We have applied this liftering process to various speech recognition problems; in particular, vowel recognition and isolated word recognition. With the liftering process, we have been able to achieve an average digit error rate of 1%, which is about half of the previously reported best results, with dynamic time warping in a speaker-independent isolated digit test.

Algorithms for recognizing strings of connected words from whole word patterns (either templates or statistical models) have advanced to the point of high efficiency and accuracy. Although the computation rate of these connected word recognition algorithms remains high, advances in VLSI hardware make even the most ambitious connected word recognition tasks practical with todays technology. The greatest impediment to the successful utilization of connected word recognizers is the difficulty in extracting reliable, robust whole word reference patterns. In the past, connected word recognizers have relied on either isolated word reference patterns (which are trivially obtained), or reference patterns derived from limited context strings of words (e.g. the middle digit from strings of 3 digits). The resulting whole word reference patterns were adequate for slow rates of speech articulation, but proved inadequate when users spoke strings of words at high rates (e.g. on the order of 200-300 words per minute). To alleviate this difficulty, a training procedure for extracting whole word patterns from naturally spoken word strings has been implemented and is described here. The training procedure is essentially a k-means loop in which a set of known word strings is segmented into individual words based on matching an initial set of word reference patterns (typically a speaker independent set of isolated word reference patterns is used). The segmented words are then used to create an updated set of word reference patterns (either via clustering methods, for templates or via statistical techniques, for word models), which are then used in the segmental loop to give an updated set of word tokens from the labelled training set. This procedure is iterated until a stable set of whole word reference patterns is obtained. The training procedure was implemented and tested in a connected digits recognition task. For this task, string accuracies (on variable length strings with from 1-7 digits) on the order of 98-99% were obtained.

In this paper we extend previous work on isolated word recognition based on hidden Markov models by replacing the discrete symbol representation of the speech signal by a continuous Gaussian mixture density. In this manner the inherent quantization error introduced by the discrete representation is essentially eliminated. The resulting recognizer was tested on a vocabulary of the 10 digits across a wide range of talkers and test conditions, and shown to have an error rate at least comparable to that of the best template recognizers and significantly lower than that of the discrete symbol hidden Markov model system. Several issues involved in the training of the continuous density models and in the implementation of the recognizer are discussed.

In this study we compared several different spectral distortion measures including the Itakura-Saito (IS), the log likelihood ratio (LLR), the likelihood ratio (LR), the cepstral (CEP), and two perceptually based distortion measures, the weighted likelihood ratio (WLR) and the weighted slope metric (WSM) distortion measures, in terms of their effects on the performance of a standard dynamic time warping (DTW) based, isolated word, speech recognizer. Two modifications of the basic forms of each measure were also investigated, namely a Bark-scale frequency warping and the incorporation of suprasegmental energy information. All distortion measures and their modifications were tested on an alpha-digit vocabulary, 4-talker, telephone recording data base. The results can be summarized as: (1) All LPC-based distortion measures performed reasonably well. The LLR and WSM distortion measures gave the highest recognition accuracy, while the IS distortion measure gave the lowest score; (2) Whereas the addition of suprasegmental energy information helped the recognition performance, the use of gain and absolute loudness degraded the performance; (3) Bark-scale frequency warping did not perform as well as its unwarped counterpart; (4) The WLR distortion measure did not perform as well as its unweighted counterpart.

Recently a new structure for isolated word recognition was proposed based on the ideas of vector quantization (VQ). In this scheme a separate VQ codebook, for each word in the vocabulary, was designed, based on a training sequence of several tokens of each word by one or more talkers. In the original implementation, the recognizer chose the word in the vocabulary whose average quantization distortion (according to its particular codebook) was minimum. In the proposed implementation, the word-based VQ's are used as a front end preprocessor to eliminate word candidates whose distortion scores are large; a DTW processor then resolves the choice among the remaining word candidates (i.e. those which are passed on by the preprocessor). Both of the above schemes work very well for small vocabularies; however the major flaw is the lack of temporal information in the word-based VQ processor. As such, as the vocabulary for recognition grows in size and complexity, the ability of the VQ processor to resolve among similar sounding words decreases dramatically, and the effectiveness of the proposed recognition structure similarly decreases. To alleviate this difficulty a technique for incorporating temporal structure into the preprocessor is also proposed. In particular, the probability density function of the time of occurrence for each vector in the codebook is estimated from the same training sequence used to derive the codebook vectors. In the recognizer, the spectral distance score of the VQ is combined with a (scaled) temporal distance score, for each frame in the word. An evaluation of the proposed recognizer showed good performance on both the digits vocabulary, and on a vocabulary of 129 airlines terms.

In this study a vector quantization (VQ) codebook was used as an efficient means of characterizing the short-time spectral features of a speaker. A set of such codebooks were then used to recognize the identity of an unknown speaker from his/her unlabelled spoken utterances based on a minimum distance (distortion) classification rule. A series of speaker recognition experiments was performed using a 100-talker (50 male and 50 female) telephone recording database consisting of isolated digit utterances. For ten random but different isolated digits, over 98% speaker identification accuracy was achieved. The effects, on performance, of different system parameters such as codebook sizes, the number of test digits, phonetic richness of the text, and difference in recording sessions were also studied in detail.

The theory of vector quantization (VQ) of linear predictive coding (LPC) coefficients has established a wide variety of techniques for quantizing LPC spectral shape to minimize overall spectral distortion. Such vector quantizers have been widely used in the areas of speech coding and speech recognition. The conventional vector quantizer utilizes only spectral shape information and essentially disregards the energy or gain term associated with the optimal LPC fit to the signal being modelled. In this paper we present a method of incorporating LPC spectral shape and energy into the codebook entries of the vector quantizer. To do this we postulate a distortion measure for comparing two LPC vectors which uses a weighted sum of an LPC shape distortion and a log energy distortion. Based on this combined distortion measure we have designed and studied vector quantizers of several sizes for use in isolated word speech recognition experiments. We have found that a fairly significant correlation exists between LPC shape and signal energy; hence a combined LPC shape plus energy vector quantizer with a given distortion requires far fewer codebook entries than one in which LPC shape and energy are quantized separately. Based on isolated word recognition tests on both a 10-digit and a 129 word airlines vocabulary, we have found improvements in recognition accuracy by using the VQ with both LPC shape and energy over that obtained using a VQ with LPC shape alone.

Automatic speech recognition has advanced to the stage where it is now possible to recognize connected strings of words (e.g. digits, letters, city names, airline terms) from a word reference set of isolated tokens of each of the words in the vocabulary. Recently an improved training technique, called embedded word training, was proposed in which reference word patterns were extracted from within connected word sequences themselves. In this investigation we extend the embedded word training procedure to handle letters of the alphabet for use in a directory listing retrieval task. By performing connected letter recognition of spoken names based on letter classes (rather than specific letters themselves) we show how reliable name recognition results can be achieved using a fairly straightforward name search procedure. We have tested the recognition system on 200 randomly chosen names (chosen from an 18,210 name directory) spoken at a normal rate by 4 talkers (3 male, 1 female) in a speaker trained mode. We have found that an 8% improvement in name recognition accuracy is obtained when using embedded letter training patterns over that obtained from isolated letter patterns alone. The overall name recognition accuracy was close to 95%.

The vast majority of commercially available isolated word recognizers use a filter bank analysis as the front end processing for recognition. To the designers of such recognizers there exists a myriad of design choices. It is not well understood how these designs choices (e.g. the number of filters, filter spacing, and post-processing of the filter bank outputs) affect recognizer performance. In this paper we present results of a performance evaluation of filter bank recognizers in a speaker-trained, isolated word recognition test using dialed-up telephone line recordings. First we studied the effects of various filter bank parameters on system performance. We designed a total of 13 filter banks including 8 uniform and 5 non-uniform filter banks. Within these 13 filter banks we also considered both slightly and highly overlapping filters. The results indicate that the best performance (highest word accuracy) on the 39 word alpha digits vocabulary, with 4 talkers, is obtained by both a 15 channel uniform filter bank and a 13 channel highly overlapping non-uniform critical band filter bank. Next we studied the effects of selected preprocessing and post-processing techniques on system performance. For this we used the non-uniform 13 channel filter bank. The results indicate that almost none of the processing techniques improved system performance; however, some techniques can potentially reduce hardware cost (computation and storage) without adversely affecting system performance. We also compared the results of the best filter banks recognizers with a conventional LPC based word recognizer on the same vocabulary. The performance of the best filter bank was approximately 4% worse than that of an 8th order LPC-based recognizer. We also studied the effect of additive wideband Gaussian noise on system performance of both the filter bank and the LPC recognizers. Gaussian white noise was added to the speech recordings at signal-to-noises ratios of from 0 to 30 dB. Recognition tests were then performed which indicated that the LPC system performance degraded faster than that of the filter banks; however, the point at which both systems have identical performance is at a signal-to-noise ratio of 6 dB.

A method for speaker independent isolated digit recognition based on modeling entire words as discrete probabilistic functions of a Markov chain is described. Training is a three part process comprising conventional methods of linear prediction coding (LPC) and vector quantization of the LPCs followed by an algorithm for estimating the parameters of a hidden Markov process. Recognition utilizes linear prediction and vector quantization steps prior to maximum likelihood classification based on the Viterbi algorithm. Vector quantization is performed by a K-means algorithm which finds a codebook of 64 prototypical vectors that minimize the distortion measure (Itakura distance) over the training set. After training based on a 1,000 token set, recognition experiments were conducted on a separate 1,000 token test set obtained from the same talkers. In this test a 3.5% error rate was observed which is comparable to that measured in an identical test of an LPC/DTW (dynamic time warping) system. The computational demand for recognition under the new system is reduced by a factor of approximately 10 in both time and memory compared to that of the LPC/DTW system. It is also of interest that the classification errors made by the two systems are virtually disjoint; thus the possibility exists to obtain error rates near 1% by a combination of the methods. In describing our experiments we discuss several issues of theoretical importance, namely: 1) Alternatives to the Baum-Welch algorithm for model parameter estimation, e.g., Lagrangian techniques; 2) Model combining techniques by means of a bipartite graph matching algorithm providing improved model stability; 3) Methods for treating the finite training data problem by modifications to both the Baum-Welch algorithm and Lagrangian techniques; and 4) Use of non-ergodic Markov chains for isolated word recognition. We note that the experiments reported here are the first in which a direct comparison is made between two conceptually different (i.e. parametric and non-parametric) methods of treating the non-stationarity problem in speech recognition by implicitly dividing the speech signal into quasi-stationary intervals.

The technique of dynamic time warping (DTW) is relied on heavily in isolated word recognition systems. The advantage of using DTW is that reliable time alignment between reference and test patterns is obtained. The disadvantage of using DTW is the heavy computational burden required to find the optimal time alignment path. Several alternative procedures have been proposed for reducing the computation of DTW algorithms. However these alternative methods generally suffer from a loss of optimality or precision in defining points along the alignment path. In this paper we propose another alternative procedure for implementing a DTW algorithm. The procedure is based on the well known class of techniques for a directed search through a grid to find the "shortest' path. An adaptive version of a directed search procedure is defined and shown to be capable of obtaining the exact DTW solution with reduced computation of distances but with increased overhead. It is shown that for machines where the time for distance computation is significantly larger than the time for combinatorics and overhead, a potential gain in speed of up to 3 to 1 can be realized with the directed search algorithm. Formal comparison of the directed search algorithm with a standard DTW method, in an isolated word recognition test, showed essentially no loss in recognition accuracy when the parameters of the directed search were selected to realize the 3 to 1 reduction in distance computation.

The "conventional" way of obtaining word reference patterns for connected word recognition systems is to use isolated word patterns, and to rely on the dynamics of the matching algorithm to account for the differences in connected speech. Connected word recognition, based on such an approach, tends to become unreliable (high error rates) when the talking rate becomes grossly incommensurate with the rate at which the isolated word training patterns were spoken. To alleviate this problem, an improved training procedure for connected word (digit) recognition is proposed in which word reference patterns from isolated occurrences of the vocabulary words are combined with word reference patterns extracted from within connected word strings to give a robust, reliable word recognizer over all normal speaking rates. In a test of the system (as a speaker trained, connected digit recognizer) with 18 talkers each speaking 40 different strings (of variable length from 2 to 5 digits), median string error rates of 0% and 2.5% were obtained for deliberately spoken strings and naturally spoken strings, respectively, when the string length was known. Using just isolated word training tokens, the comparable error rates were 10% and 11.3% respectively.

It has long been known that one of the key factors in determining the accuracy of isolated word recognition systems is the size and/or complexity of the vocabulary. Although most practical isolated word recognizers use small vocabularies (on the order of 10 to 50 words), there are many applications which require medium to large size vocabularies (e.g. airlines reservation and information, data retrieval etc). It is the purpose of this paper to discuss the problems associated with speaker-trained recognition of a large vocabulary (1109 words) of words. It is shown that the practicability of using large vocabularies for isolated word vocabularies is doubtful, both because of the problems in training the system, and because of the difficulty for the user to learn and remember the vocabulary words for any significant size vocabulary. The importance of studying large word vocabularies for recognition lies in the flexibility it provides for understanding the effects of vocabulary size and complexity on recognition accuracy for both small and medium size vocabularies. By constructing subsets of the total vocabulary for recognition, we show that a judicious choice of words can lead to significantly better recognition accuracy than by poor choice of the words in the subset. We show that for each doubling of the size of the vocabulary, the recognition accuracy tends to decrease by a fixed amount, which is different for each talker.

A digital-based isolated word recognition system has been implemented in a module of dedicated hardware that uses a microprocessor and programmable digital signal processing circuitry. The recognizer is based upon the minimum prediction residual principle of Itakura. The recognition algorithm has been developed and tested on a general-purpose minicomputer and array processor, where it has been shown to be suitable for several recognition tasks. The recognition hardware consists of an Intel 8086 16-bit microprocessor operating in parallel with a digital speech processing peripheral (DSPP) tailored to the algorithm. The microprocessor performs the supervisory and decision operations; the DSPP performs the 200,000T + 4,500N multiply-add operations (and associated data transfers) associated with the recognition of a word of duration T sec from an N word vocabulary with 1 template per word. The recognizer is compact (board area of 250 sq. in.) and inexpensive (commercial component cost of about $1200 for 40 word templates).

The technique of dynamic time warping has proven itself reliable and robust for a wide variety of isolated word recognition tasks. Recently extensions of the algorithm have been investigated for application to the problem of connected word recognition. In this paper a level building technique is proposed for optimally aligning a test pattern, consisting of a sequence of connected words, with a sequence of isolated word reference patterns. This algorithm is shown to be significantly more efficient than the one proposed by Sakoe while solving the exact same problem. Implementation parameters for the level building algorithm are presented and the effectiveness of the proposed algorithm for connected digit recognition is experimentally verified.

One of the major drawbacks of the standard pattern recognition approach to isolated word recognition is that poor performance is generally achieved for word vocabularies with acoustically similar words. This poor performance is related to the pattern similarity (distance) algorithms that are generally used in which a global distance between the test pattern and each reference pattern is computed. Since acoustically similar words are, by definition, globally similar, it is difficult to reliably discriminate such words, and a high error rate is obtained. By modifying the pattern similarity algorithm so that the recognition decision is made in two passes, improvements in discriminability among similar words can be achieved. In particular, on the first pass the recognizer provides a set of global distance scores which are used to decide a class (or a set of possible classes) in which the spoken word is estimated to belong. On the second pass a locally weighted distance is used to provide optimal separation among words in the chosen class (or classes) and the recognition decision is made on the basis of these local distance scores. For a highly complex vocabulary (letters of the alphabet, digits, and 3 command words) recognition improvements of from 3 to 7 percent were obtained using the two-pass recognition strategy.

A speech recognition system is described for recognizing isolated words from reference templates created by concatenating demisyllables from a corpus of about 1000 demisyllables. The composition (in terms of demisyllables) of each reference word is specified in a lexicon with one or more entries for each word of the vocabulary. Experiments were carried out, using a 100-word vocabulary, to investigate the usefulness of such a representation and the effect on performance of some simple modifications in demisyllable specification and durations of reference patterns. Recognition accuracy of 97.6% was obtained using 132 reference templates for the 100-word vocabulary.

The art and science of speech recognition have been advanced to the state where it is now possible to communicate reliably with a computer by speaking to it in a disciplined manner using a vocabulary of moderate size. It is the purpose of this paper to outline two aspects of speech-recognition research. First, we discuss word recognition as a classical pattern-recognition problem and show how some fundamental concepts of signal processing, information theory, and computer science can be combined to give us the capability of robust recognition of isolated words and simple connected word sequences. We then describe methods whereby these principles, augmented by modern theories of formal language and semantic analysis, can be used to study some of the more general problems in speech recognition. It is anticipated that these methods will ultimately lead to accurate mechanical recognition of fluent speech under certain controlled conditions.

The log likelihood measure has been widely used in speech recognition for comparing speech signals. Recently it has been proposed as a measure for assessing the quality of coded speech. In this paper we present an interpretation of the log likelihood ratio measure within the theoretical framework of a waveform coder distortion model.

Several variations on algorithms for dynamic time warping have been proposed for speech processing applications. In this paper two general algorithms that have been proposed for word spotting and connected word recognition are studied. These algorithms are called the fixed range method and the local minimum method. The characteristics and properties of these algorithms are discussed. It is shown that, in several simple performance evaluations, the local minimum method performed considerably better then the fixed range method. Explanations of this behavior are given and an optimized method of applying the local minimum algorithm to word spotting and connected word recognition is described.

A connected digit recognizer is proposed in which a set of isolated word templates is used as reference patterns and an unconstrained dynamic time warping algorithm is used to literally spot the digits in the string. Segmentation boundaries between digits are obtained as the termination point of the dynamic path from the previous time warp. A region around the boundary is searched for the optimum starting point for the succeeding digit. At each stage the recognizer keeps track of a set of candidate digit strings for each test string. The string with the smallest accumulated distance is used as the preliminary string estimate. For variable length digit strings of from 2 to 5 digits (where the recognizer was not told the length of the string), word error rates of about 2-3% and string error rates on the order of 8% were obtained for both speaker dependent and speaker independent systems.

In this paper we describe a speaker trained, voice controlled, repertory dialer system. The main elements of the system include: 1. A real-time speech analyzer that detects the presence of speech on the input line, and analyzes the speech to give features appropriate for a word recognizer. 2. An isolated word recognizer that decides which of a set of words was spoken. 3. A voice response system to provide spoken commands to the user to guide the use of the repertory dialer system. 4. A dialer (simulated) to outpulse the desired telephone number. The repertory dialer system is implemented on a minicomputer with a high speed array processor performing the real-time operations. The vocabulary for the system consists of 7 command words, 10 digits, and any number of names up to some specified maximum Recognition is performed on one or more subsets of the vocabulary, depending on fine state of the system. To train the system the user is requested to speak each of the vocabulary words twice to provide reference templates for the system. Following training, the system can dial the telephone number corresponding to any name in the repertory, or it can dial a 4 digit telephone extension spoken as an isolated string of digits. The system was tested extensively by 6 talkers (3 male, 3 female - 3 of whom were naive and 3 experienced users) over a three week period. A total of 4620 words were spoken and during the course of the test there were no recognition errors. A request for a repeat of a spoken word occurred about 2% of the time. These tests demonstrate the reliability and robustness of this voice repertory dialer system.

A speaker independent, isolated word recognition system is proposed which is based on the use of multiple templates for each word in the vocabulary. The word templates are obtained from a statistical clustering analysis of a large data base consisting of 100 replications of each word (i.e. once by each of 100 talkers). The recognition system, which uses telephone recordings, is based on an LPC analysis of the unknown word, dynamic time warping of each reference template to the unknown word (using the Itakura LPC distance measure), and the application of a K-nearest neighbor (KNN) decision rule to lower the probability of error. Results are presented on two test sets of data which show error rates that are comparable to, or better than, those obtained with speaker trained, isolated word recognition systems.

Recent work at Bell Laboratories has demonstrated the utility of applying sophisticated pattern recognition techniques to obtain a set of speaker independent word templates for an isolated word recognition system [1,2]. In these studies, it was shown that a careful experimenter could guide the clustering algorithms to choose a small set of templates that were representative of a large number of replications for each word in the vocabulary. Subsequent word recognition tests verified that the templates chosen were indeed representative of a fairly large population of talkers. Given the success of this approach, the next important step is to investigate fully automatic techniques for clustering multiple versions of a single word into a set of speaker independent word templates. Two such techniques are described in this paper. The first method uses distance data (between replications of a word) to segment the population into stable clusters. The word template is obtained as either the cluster minimax, or as an averaged version of all the elements in the cluster. The second method is a variation of the one described by Rabiner [3] in which averaging techniques are directly combined with the nearest neighbor rule to simultaneously define both the word template (i.e. the cluster center) and the elements in the cluster. Experimental data shows the first method to be superior to the second method when 3 or more clusters per word are used in the recognition task.

A wide variety of distance measures for comparing sets of LPC coefficients have been proposed. The most popular one has been the log likelihood ratio test as proposed by Itakura [1]. Although this measure is both computationally efficient, and theoretically sound, de Souza, in a recent paper [2], found that the measured distribution of LPC distances was significantly different from the one predicted by theory. De Souza then proposed alternative statistical tests for measuring LPC distances. In this paper we present both theoretical and experimental results which show that there is, in general, excellent agreement between theory and practice when using the log likelihood ratio to measure LPC distance. Included in the paper are discussions of the effects of LPC analysis method (covariance or auto-correlation), the use of fixed pre-emphasis, and the effects of additive, uncorrelated noise on the distributions.

In this paper we investigate the performance of three well known system identification methods based on an FIR (finite impulse response) model of the system. The methods will be referred to in this paper as the least squares analysis (LSA) method, the least mean Squares adaptation algorithm (LMS) and the short-time spectral analysis (SSA) procedure. Our particular interest in this paper concerns the performance of these algorithms in the presence of high noise levels and in situations where the input signal is bandlimited. Both white and nonwhite random noise signals as well as speech signals are used as test signals to measure the performance of the system identification techniques, Quantitative results in terms of an accuracy measure of system identification are presented and a simple analytical model is used to explain the measured results.

A comparative performance study of seven pitch detection algorithms was conducted. A speech data base, consisting of eight utterances spoken by 3 males, 3 females, and 1 child was constructed. Telephone, close talking microphone, and wideband recordings were made of each of the utterances. For each of the utterances in the data base a "standard" pitch contour was semiautomaticallly measured using a highly sophisticated interactive pitch detection program. The "standard" pitch contour was then compared with the pitch contour that was obtained from each of the seven programmed pitch detectors. The algorithms used in this study were (1) a center clipping, infinite-peak clipping, modified autocorrelation method, (2) the cepstral method, (3) the SIFT method, (4) the parallel processing time domain method, (5) the data reduction method, (6) a spectral flattening LPC method, and (7) the AMDF method. A set of measurements was made on the pitch contours to quantify the various types of errors which occur in each of the above methods. Included among the error measurements were the average and standard deviation of the error in pitch period during voiced regions, the number of gross errors in the pitch period, and the number of voiced-unvoiced classification errors. For each of the error measurements, the individual pitch detectors could be rank ordered as a measure of their relative performance as a function of recording condition, and pitch range of the various speakers. Results are presented on rankings based on one category of errors.

In this paper we discuss a number of issues concerning the design of optimal decimators, and interpolators and their application to narrow band filtering. We show that a design which is optimized for minimum storage is essentially optimum in terms of its number of required multiplications and additions as well. We then discuss a practical scheme for efficiently implementing decimators and interpolators.

This paper describes an implementation of a speaker independent system which can recognize connected digits. The overall recognition system consists of two separate but inter-related parts. The function of the first part of the system is to segment the digit string into the individual digits which comprise the string; the second part of the system then recognizes the individual digits based on the results of the segmentation. To evaluate the accuracy of the system in segmenting and recognizing digit strings a series of experiments was conducted. Using high quality recordings from a soundproof booth the segmentation accuracy was found to be about 99%, and the recognition accuracy was about 91% across 10 speakers (5 male, 5 female). With recordings made in a noisy computer room the segmentation accuracy remained close to 99%, and the recognition accuracy was about 87% across another group of 10 speakers (5 male, 5 female).

When speech is coded using a differential pulse-code modulation system with an adaptive quantizer, the digital code words exhibit considerable variation among all quantization levels during both voiced and unvoiced speech intervals. However, because of limits on the range of step sizes, during silent intervals the code words vary only slightly among the smallest quantization steps. Based on this principle, a simple algorithm for locating the beginning and end of a speech utterance has been developed. This algorithm has been tested in computer simulations and has been constructed with standard integrated circuit technology.