In this paper, we describe our approach for bootstrapping statistical language models for spoken dialog systems using indomain Web data and utterances collected from previous applications. The approach is based on the idea of stitching conversational templates with the predicate and arguments extracted from the Web pages using semantic role labeling, to generate conversational style utterances. The conversational templates represent the task-independent portions of user utterances and can be built by hand, or learned from utterances collected from other domain applications. Experiments have shown that, stitching with both types of conversational templates have resulted in significantly better ASR word accuracy. Furthermore, the new language model bootstrapping approach can be combined with unsupervised and active learning to improve word accuracy even with very little in-domain transcribed data

This paper describes the AT&T WATSON real-time speech recognizer, the product of several decades of research at AT&T. The recognizer handles a wide range of vocabulary sizes and is based on continuous-density hidden Markov models for acoustic modeling and finite state networks for language modeling. The recognition network is optimized for efficient search. We identify the algorithms used for high-accuracy, real-time and low-latency recognition. We present results for small and large vocabulary tasks taken from the AT&T VoiceTone/sup /spl reg// service, showing word accuracy improvement of about 5% absolute and real-time processing speed-up by a factor between 2 and 3.

The use of boosting for call classification in spoken language understanding is described in this paper. An extension to the AdaBoost algorithm is presented that permits the incorporation of prior knowledge of the application as a means of compensating for the large dependence on training data. We give a convergence result for the algorithm, and we describe experiments on four datasets showing that prior knowledge can substantially improve classification performance.

A key challenge in rapidly building spoken natural language dialog applications is minimizing the manual effort required in transcribing and labeling speech data. This task is not only expensive but also time consuming. We present a novel approach that aims at reducing the amount of manually transcribed in-domain data required for building automatic speech recognition (ASR) models in spoken language dialog systems. Our method is based on mining relevant text from various conversational systems and Web sites. An iterative process is employed where the performance of the models can be improved through both unsupervised and active learning of the ASR models. We have evaluated the robustness of our approach on a call classification task that has been selected from AT&T VoiceTone/sup SM/ customer care. Our results indicate that with unsupervised learning it is possible to achieve a call classification performance that is only 1.5% lower than the upper bound set when using all available in-domain transcribed data.

WebTalk is an intelligent agent that automatically mines company websites to create interactive spoken and chat-based dialog systems. In this paper, we briefly describe the technologies that WebTalk utilizes for extracting task knowledge from websites and answering questions specified in natural language. We elaborate on an evaluation of the question answering component of WebTalk. In particular, we compare this component with a deployed hand-crafted question answering agent. A detailed evaluation analysis is presented.

The performance of large vocabulary speech recognizers often varies depending on the input speech and the quality of the trained models. The particular attributes that cause recognition errors are a research area that has not been well studied. This paper addresses this issue from a robustness perspective using a large amount of field data collected from natural language dialog services. In particular, we present a method for tracking time-varying or nonstationary extraneous events, such as music, background noise, etc. We show that this measure is a better predictor of recognition errors than a standard measure of stationary signal-to-noise ratio (SNR). Combining the two measures provides a data selection algorithm for detecting problematic speech.

State-of-the-art spoken language understanding (SLU) systems are trained using human-labeled utterances, preparation of which is labor intensive and time consuming. Labeling is an error-prone process due to various reasons, such as labeler errors or imperfect description of classes. Thus, usually a second (or maybe more) pass(es) of labeling is required in order to check and fix the labeling errors and inconsistencies of the first (or earlier) pass(es). In this paper, we check the effect of labeling errors for statistical call classification and evaluate methods of finding and correcting these errors by checking minimum amount of data. We describe two alternative methods to speed up the labeling effort, one is based on the confidences obtained from a prior model and the other completely unsupervised. We call the labeling process employing one of these methods as active labeling. Active labeling aims to minimize the number of utterances to be checked again by automatically selecting the ones that are likely to be erroneous or inconsistent with the previously labeled examples. Although very same methods can be used as a postprocessing step to correct labeling errors, we only consider them as part of the labeling process. We have evaluated these active labeling methods using a call classification system used for AT&T natural dialog customer care system. Our results indicate that it is possible to find about 90% of the labeling errors or inconsistencies by checking just half the data.

Data collection and annotation are major bottlenecks in rapid development of accurate syntactic and semantic models for natural-language dialogue systems. In this paper we show how human knowledge can be used when designing a language understanding system in a manner that would alleviate the dependence on large sets of data. In particular, we extend BoosTexter, a member of the boosting family of algorithms, to combine and balance hand-crafted rules with the statistics of available data. Experiments on two voice-enabled applications for customer care and help desk are presented.

The design and implementation of the AT&T Communicator mixed-initiative spoken dialog system is described.The Communicator project, sponsored by DARPA and launched in 1999, is a multi-year multi-site project on advanced spoken dialog systems research.The main focus of this paper is on issues related to the design of mixed-initiative systems.In addition to describing our architecture and implementation of the complex travel task, the paper reports on some preliminary evaluation results.

This paper describes our progress towards building a telephony-based spoken dialogue system for workshop/conference services. A mixed-initiative dialogue system has been developed that is engineered to o er users natural interaction with the system, ease-of-use and robustness towards ambiguous requests and machine errors. A prototype system, known as W99, is described in this paper

Hidden Markov models (HMMs) for automatic speech recognition rely on high dimensional feature vectors to summarize the short-time properties of speech. Correlations between features can arise when the speech signal is nonstationary or corrupted by noise. We investigate how to model these correlations using factor analysis, a statistical method for dimensionality reduction. Factor analysis uses a small number of parameters to model the covariance structure of high dimensional data. These parameters can be chosen in two ways: (1) to maximize the likelihood of observed speech signals, or (2) to minimize the number of classification errors. We derive an expectation-maximization (EM) algorithm for maximum likelihood estimation and a gradient descent algorithm for improved class discrimination. Speech recognizers are evaluated on two tasks, one small-sized vocabulary (connected alpha-digits) and one medium-sized vocabulary (New Jersey town names). We find that modeling feature correlations by factor analysis leads to significantly increased likelihoods and word accuracies. Moreover, the rate of improvement with model size often exceeds that observed in conventional HMM's.

This paper addresses the general problem of connected digit recognition in the telecommunication environment. In particular, we focus on a task of recognizing digits when embedded in a natural spoken dialog. Two different design strategies are investigated: keyword detection of word spotting, and large-vocabulary continuous speech recognition. We characterize the potential benefits and describe the main components of each design method, including acoustic and language modeling, training and utterance verification. Experimental results on a subset of a database that includes customers responses to the open-ended prompt "How may I help you?" are presented.

Explicit expressions for the second-order statistics of cepstral components representing clean and noisy signal waveforms are derived. The noise is assumed additive to the signal, and the spectral components of each process are assumed statistically independent complex Gaussian random variables. The key result developed here is an explicit expression for the cross-covariance between the log-periodograms of the clean and noisy signals. In the absence of noise, this expression is used to show that the covariance matrix of cepstral components representing N signal samples, is a fixed signal independent matrix, which approaches a diagonal matrix at a rate of 1/N. In addition, the cross-covariance expression is used to develop an explicit linear minimum mean square error estimator for the clean cepstral components given noisy cepstral components. Recognition results on the English digits using the fixed covariance and linear estimator are presented.

The performance of an automatic speech recognizer degrades when there exists an acoustic mismatch between the training and the testing conditions in the data. Though it is certain that the mismatch is nonlinear, its exact form is unknown. Tackling the problem of nonlinear mismatches is a difficult task that has not been adequately addressed before. We develop an approach that uses nonlinear transformations in the stochastic matching framework to compensate for acoustic mismatches. The functional form of the nonlinear transformation is modeled by neural networks. We develop a new technique to train neural networks using the generalized EM algorithm. This technique eliminates the need for stereo databases, which are difficult to obtain in practical applications. The new technique is data-driven and hence can be used under a wide variety of conditions without a priori knowledge of the environment. Using this technique, we show that we can provide improvement under various types of acoustic mismatch; in some cases a 72% reduction in word error rate is achieved.

Explicit expressions for the second order statistics of cepstral components representing clean and noisy signal waveforms are derived. The noise is assumed additive to the signal, and the spectral components of each process are assumed statistically independent complex Gaussian random variables. The key result developed here is an explicit expression for the cross-covariance between the log-spectra of the clean and noisy signals. In the absence of noise, this expression is used to show that the covariance matrix of cepstral components representing a vector of N signal samples, approaches a fixed, signal independent, diagonal matrix at a rate of 1/N/sup 2/. In addition, the cross-covariance expression is used to develop an explicit linear minimum mean square error estimator for the clean cepstral components given noisy cepstral components. Recognition results on the ten English digits using the fixed covariance and linear estimator are presented.

A system for discriminative feature and model design is presented for automatic speech recognition. Training based on minimum classification error with a single objective function is applied for designing a set of parallel networks performing feature transformation and a set of hidden Markov models performing speech recognition. This paper compares the use of linear and non-linear functional transformations when applied to conventional recognition features, such as spectrum or cepstrum. It also provides a framework for integrated feature and model training when using class-specific transformations. Experimental results on telephone-based connected digit recognition are presented. 1. INTRODUCTION Improving the performance of hidden Markov model (HMM) based automatic speech recognition (ASR) systems has been a central issue that has dominated the entire field of speech recognition during the past two decades. One effort to improving HMMs has been by extending the training paradigm beyo...

Utterance verification represents an important technology in the design of user-friendly speech recognition systems. It involves the recognition of keyword strings and the rejection of nonkeyword strings. This paper describes a hidden Markov model-based (HMM-based) utterance verification system using the framework of statistical hypothesis testing. The two major issues on how to design keyword and string scoring criteria are addressed. For keyword verification, different alternative hypotheses are proposed based on the scores of antikeyword models and a general acoustic filler model. For string verification, different measures are proposed with the objective of detecting nonvocabulary word strings and possibly erroneous strings (so-called putative errors). This paper also motivates the need for discriminative hypothesis testing in verification. One such approach based on minimum classification error training is investigated in detail. When the proposed verification technique was integrated into a state-of-the-art connected digit recognition system, the string error rate for valid digit strings was found to decrease by 57% when setting the rejection rate to 5%. Furthermore, the system was able to correctly reject over 99.9% of nonvocabulary word strings.

This paper focuses on one aspect of achieving flexible speech recognition, namely, improving the ability to cope with naturally spoken utterances through discriminative utterance verification. We propose an algorithm for training utterance verification systems based on the minimum verification error (MVE) training framework. Experimental results on speaker-independent connected digits, show a significant improvement in verification accuracy when the discriminant function used in MVE training is made consistent with the confidence measure used in utterance verification. At a 10% rejection rate, for example, the new proposed method reduces the string error rate by a further 22.7% over our previously reported results in which the MVE based discriminative training was not incorporated.

An utterance verification method based on minimum verification error training is presented. In a two-stage process, the recognition hypothesis produced by an HMM-based speech recognizer is verified using a set of verification-specific models that are independent of the models used in the recognition process. The verification models are trained using a discriminative training procedure that seeks to minimize the verification error by simultaneously maximizing the rejection of non-keywords and misrecognized keywords while minimizing the rejection of correctly recognized keywords. This method is evaluated on a connected digit recognition task with a null grammar. The baseline string error rate for this task was 4.85%. At 5% rejection of valid strings, the string error rate decreased to 2.70% using the proposed verification method. The corresponding performance on non-keyword speech was a rejection rate of over 99.0%.

Acoustic mismatch encountered in various training and testing conditions of hidden Markov model (HMM) based systems often causes severe degradation in speech recognition performance. For telephone based speech recognition tasks, acoustic mismatch can arise from various sources, such as variations in telephone handsets, ambient noise, and channel distortions. This paper presents three techniques for blind channel equalization, namely, cepstral mean subtraction (CMS), signal bias removal (SBR) and hierarchical signal bias removal (HSBR). Experimental results on various connected digits databases show a reduction in the digit error rate by 16%, 21%, and 28% when employing CMS, SBR, and HSBR, respectively. Our results also demonstrate that the HSBR technique outperforms SBR and CMS on every sub-data collection and exhibits consistent improvements even for short utterances.

An acoustical mismatch between the training and the testing conditions of hidden Markov model (HMM)-based speech recognition systems often causes a severte degradation in the recognition performance. In telephone speech recognition, for example, undesirable signal components due to ambient noise and channel distortion, as well as due to different variations of telephone handsets render the recognizer unusable for real- world applications. This paper presents a signal bias removal (SBR) method based on maximum likelihood1 estimation for the minimization of these undesirable effects. The proposed method is readily applicable in various architectures, i.e., dis- crete (vector-quantization based), semicontinuous and continuous density HMM. In this paper, the SBR method, integrated into a discrete density HMM, is applied to telephone speech recognition where the contamination due to extraneous signal components is assumed to be unknown. To enable real-time implementation, a sequential method for the estimation of the bias is presented. Experimental results for speaker-independent connected digit recognition show a reduction in the per digit error rate by up to 41% and 14% during mismatched and matclhed training and testing conditions, respectively.

Utterance verification represents an important technology in the design of user-friendly speech recognition systems. This paper addresses the issue of robustness in utterance verification. Four different approaches to robustness have been investigated: a string based likelihood measure for the detection of non-vocabulary words and "putative" errors, a signal bias removal method for channel normalization, on-line adaptation technique for achieving desirable trade-off between false rejection and false alarms, and a discriminative training method for the minimization of the expected string error rate. When these techniques were all integrated into a state-of-the-art connected digit recognition system, the string error rate was found to decrease by up to 57% at a rejection rate of 5%. For non-vocabulary word strings, the proposed utterance verification system rejected over 99.9% of extraneous speech.

Connected digit speech recognition in the telephone network is becoming increasingly more important as the demand for speech technology becomes widespread. In the past few years, several highly successful techniques for recognizing spoken connected digit strings have been proposed. Although these techniques have been applied to non-telephone based speech [e.g. Texas Instruments database], they have produced high recognition performance. Further, similar levels of performances have been demonstrated using discrete density and continuous density based hidden Markov models (HMMs). The success of the vector quantized (VQ) modeling approach, in particular, is encouraging and rather important from the viewpoint of computational efficiency. This paper presents a study of connected digit recognition on telephone network based data using VQ HMMs. We investigate several factors affecting the error rate of VQ HMMs-such as maximum mutual information (MMI) training, sender modeling, and codebook size-and measure their contributions to recognition accuracy. The model architecture, number of states and transitions, is also optimized and its contribution to overall performance discussed.>

A speech signal transmitted through a telephone channel often encounters variable conditions which significantly deteriorate the performance of state-of-the-art HMM-based speech recognition systems. Undesirable components due to ambient noise and channel interference, as well as different sound pick-up equipments, render the recognizer unsuitable for real-world applications. This paper presents a signal bias removal method based on the maximum likelihood estimation for the minimization of these undesirable effects. The proposed method, integrated into a discrete density HMM, is applied to telephone speech recognition where the contamination due to ambient noise and channel distortion are assumed to be unknown. Experimental results are presented for speaker-independent connected digit recognition which demonstrate a reduction in the word error rate by up to 40%.>

Neural tree networks (NTNs) provide an efficient technique for pattern classification. They combine the concept of decision trees with neural networks (NNs). An efficient algorithm is presented for growing NTNs through analysis of the relative confusion among the classes. The NTN is tested on 36 phonemes extracted from the TIMIT database. Results show that this implementation with five hidden neurons at each tree node grows to 8 levels and scores 58.6% correct classification, as opposed to a NN with best performance of 52.4% using 130 hidden neurons. In addition to advantages in computational complexity and recognition performance, the NTN is found to provide important phonemic correlations which are known to exist in the human auditory system.>

The authors describe an efficient procedure for acoustic-to-articulatory parameter mapping using neural networks. An assembly of multilayer perceptrons, each designated to a specific region in the articulatory space, is used to map acoustic parameters of the speech into tract areas. The training of this model is executed in two stages; in the first stage a codebook of suitably normalized articulatory parameters is used and in the second stage real speech data are used to further improve the mapping. In general, acoustic-to-articulatory parameter mapping is nonunique; several vocal tract shapes can result in identical spectral envelopes. The model accommodates this ambiguity. During synthesis, neural networks are selected by dynamic programming using a criterion that ensures smoothly varying vocal tract shapes while maintaining a good spectral match.>

The authors describe the training and use of a multilayer perceptron (MLP) which performs a mapping from the spectra of vowels and nasal consonants, using examples spoken by a single speaker, to sets of area parameters for use in the vocal-tract-modelling filter of a speech synthesizer. Different MLP structures have been investigated using as input data either PARCOR coefficients or sample values of the spectrum. The trained MLP can be used to estimate the driving parameters for speech synthesis from natural utterances using this restricted phoneme set.>