Coping with Information Overload is a major challenge of the 21stcentury. Huge volumes and varieties of multilingual data must be processed to extract salient information. Previous research has addressed automatic characterization of streaming content. However, information includes both content and associated meta-data, which humans deal with as a gestalt but computer systems often treat separately. Random attributed graphs provide an effective means to characterize and draw inferences from large volumes of language content plus associated meta-data. This paper describes these methods and their utility, with experimental proof-of-concept on the Switchboard and Enron corpora.

The goal of this research is to infer traits about groups of people from their turn-taking behavior in natural conversation. These traits are latent attributes in a social network, whose relative frequencies we estimate from content-derived metadata. Our approach is to train statistical models of turn-taking behavior using automatic labels of speech activity, and measure the association of these models with socially correlated traits. We experimentally evaluate these ideas using the Switchboard-1 speech corpus, which provides speech content and metadata associated with each speaker, such as gender, age and education, as well as inferred social correlates such as willingness to participate and initiate. We show that population proportions of these socially correlated externals can be predicted with a root mean-squared error of approximately 0.1 across all mixture proportions.

In processing large volumes of speech and language data, we are often interested in the distribution of languages, speakers, topics, etc. For large data sets, these distributions are typically estimated at a given point in time using pattern classification technology. Such estimates can be highly biased, especially for rare classes. While these biases have been addressed in some applications, they have thus far been ignored in the speech and language literature. This neglect causes significant error for low-frequency classes. Correcting this biased distribution involves exploiting uncertain knowledge of the classifier error patterns. The Metropolis-Hastings algorithm allows us to construct a Bayes estimator for the true class proportions. We experimentally evaluate this algorithm for a speaker recognition task. In this experiment, the Bayes estimator reduces maximum RMSE by a factor of five. Performance is furthermore more consistent, with range of RMSE reduced by a factor of 4.

In processing large volumes of speech and language data, we are often interested in the distribution of languages, speakers, topics, etc. For large data sets, these distributions are typically estimated at a given point in time using pattern classification technology. It is well known that such estimates can be highly biased, especially for rare classes. While these biases have been addressed in some applications, they have thus far been ignored in the speech and language literature. This neglect causes significant error for low-frequency classes. Correcting this biased distribution involves exploiting uncertain knowledge of the classifier error patterns. We describe a numerical method, the Metropolis-Hastings (M-H) algorithm, which provides a Bayes estimator for the distribution. We experimentally evaluate this algorithm for a speaker recognition task, demonstrating a fivefold reduction in root mean squared error.

Spoken dialog systems are becoming increasingly common in deployed services. These systems are not perfect, and are often deployed "open-loop" - lacking in systematic procedures for diagnosing problems and for making improvements. In order to target improvements where they will have the biggest impact, two things are needed: first, methods and tools for detailed analysis of a data feed of call logs and customer audio; second, an interactive tool for presenting an intuitive view of the results to those responsible for the application. We discuss the paradigm and an implementation through which we are able to close the loop between system execution and system evolution by providing an empirical dialog trajectory analysis represented via a stochastic finite state machine. Novel graph analysis algorithms are introduced for change detection, compression and pruning for display, based on user-interface objectives.

In this paper we investigate the utility of three aspects of named entity processing: detection, localization and value extraction. We corroborate this task categorization by providing examples of practical applications for each of these subtasks. We also suggest methods for tackling these subtasks, giving particular attention to working with speech data. We employ Support Vector Machines to solve the detection task and show how localization and value extraction can successfully be dealt with using a combination of grammar-based and statistical methods. 1.

In speech and language processing, information about the errors made by a learning system is commonly used to assess and improve its performance. Because of high computational complexity, the context of the errors is usually either ignored, or exploited in a simplistic form. The complexity becomes tractable, however, for phone recognition because of the small lexicon. For phonebased systems, an exhaustive modeling of local context is possible. Furthermore, recent research studies have shown phone recognition to be useful for several spoken language processing tasks. In this paper, we present a mechanism which learns patterns of context-sensitive errors from ASR-output aligned with the “true” phone transcriptions. We also show how this information, encoded as a context-sensitive weighted transducer, can provide a modest improvement to phone recognition accuracy even when no transcriptions are available for the domain of interest.

The next generation of voice-based user interface technology will enable easy-to-use automation of new and existing communication services, achieving a more natural human-machine interaction. By natural, we mean that the machine understands what people actually say, in contrast to what a system designer expects them to say. This approach is in contrast with menu-driven or strongly-prompted systems, where many users are unable or unwilling to navigate such highly structured interactions. AT&Ts How May I Help You? (HMIHY)(sm) technology shifts the burden from human to machine wherein the system adapts to peoples language, as contrasted with forcing users to learn the machins jargon. We have developed algorithms which learn to extract meaning from fluent speech via automatic acquisition and exploitation of salient words, phrases and grammar fragments from a corpus. In this talk I will describe the speech, language and dialog technology underlying HMIHY, plus experimental evaluation on live customer traffic from AT&T's national deployment for customer careAllen Gorin is the Head of the Speech Interface Research Department at AT&T Laboratories, with long-term research interests focusing on machine learning methods for spoken language understanding. In recent years, he has led a research team in applying speech, language and dialog technology to AT&Ts "How May I Help You?" (HMIHY) (sm) service, which has been deployed nationally for long distance customer care. He was awarded the 2002 AT&T Science and Technology Medal for his research contributions to spoken language understanding for HMIHYHe received the B.S. and M.A. degrees in Mathematics from SUNY at Stony Brook, and the Ph.D. in Mathematics from the CUNY Graduate Center in 1980. From 1980-83 he worked at Lockheed investigating algorithms for target recognition from time-varying imagery. In 1983 he joined AT&T Bell Labs where he was the Principal Investigator for AT&T's ASPEN project within the DARPA Strategic Computing Program, investigating parallel architectures and algorithms for pattern recognition. In 1987, he was appointed a Distinguished Member of the Technical Staff. In 1988, he joined the Speech Research Department at Bell Labs. He has served as a guest editor for the IEEE Transactions on Speech and Audio, and was a visiting researcher at the ATR Interpreting Telecommunications Research Laboratory in Japan. He is a member of the Acoustical Society of America, Association for Computational Linguistics and an IEEE Senior MemberHome page for Allen Gorin: http://www.research.att.com/info/algor.

State-of-the-art speech recognition systems are trained using transcribed utterances, preparation of which is labor intensive and time-consuming. In this paper, we describe a new method for reducing the transcription effort for training in automatic speech recognition (ASR). Active learning aims at reducing the number of training examples to be labeled by automatically processing the unlabeled examples, and then selecting the most informative ones with respect to a given cost function for a human to label. We automatically estimate a confidence score for each word of the utterance, exploiting the lattice output of a speech recognizer, which was trained on a small set of transcribed data. We compute utterance confidence scores based on these word confidence scores, then selectively sample the utterances to be transcribed using the utterance confidence scores. In our experiments, we show that we reduce the amount of labeled data needed for a given word accuracy by 27%. 1.

Our research concerns spoken language understanding within the domain of automated telecommunication services. In the recent papers we presented a new methodology for training of statistical language models for recognition and understanding of utterances from large corpora of phone sequences obtained as the output of a task-independent ASR-system. The advantage of this strategy compared to the traditional word-based strategy is that we don't have to manually transcribe large amounts of data in order to extract acoustic morphemes to train the classifier. Since the baseline strategy suffered high False Rejection Rates caused by finding no acoustic morphemes in the test data, we describe in this paper how approximate matching can be incorporated in the Bayes-classifier to reduce FRR. The experiments are evaluated for "How May I Help You?"-task.

A process for segmenting 2-speaker telephone conversations by speaker with no prior speaker models is described and evaluated. The process consists of an initial segmentation using acoustic change and pause detection, segment clustering, and iterative modeling of segment clusters and resegmentation. The technique has been evaluated on 6, approximately 3 min long, customer care conversations. The technique does not resolve short ( 2 secs) or overlapping segments very well, but is capable of detecting longer segments ( 4 secs) with miss rates of the order of 10% and confusion rates 2% or less.

A natural language spoken dialog system includes a large vocabulary automatic speech recognition (ASR) engine, whose output is used as the input of a spoken language understanding component. Two challenges in such a framework are that the ASR component is far from being perfect and the users can say the same thing in very different ways. So, it is very important to be tolerant to recognition errors and some amount of orthographic variability. In this paper, we present our work on developing new methods and investigating various ways of robust recognition and understanding of an utterance. To this end, we exploit word-level confusion networks (sausages), obtained from ASR word graphs (lattices) instead of the ASR 1-best hypothesis. Using sausages with an improved confidence model, we decreased the calltype classification error rate for AT&T's How May I Help You (HMIHY ) natural dialog system by 38%.

Consider a conversational speech system incorporating spoken language understanding (SLU) and dialog manager (DM) modules. A prerequisite of natural dialog is that understanding takes place in context, which in turn necessitates a sharing of task knowledge between the two modules. However, because they are engaged in different activities it is often the case that the task knowledge representation is different for each of them, an undesirable duplication. In this paper we consider AT&T’s How May I Help You? SM natural dialog system. We present a method for automatically inducing the task knowledge representation used for spoken language understanding from that used for dialog management. Also, the context information needed by the SLU module for understanding each utterance is generated automatically by the DM from its own knowledge representation and the current dialog situation. This enables understanding-incontext to be implemented while avoiding the duplication involved in creating and maintaining separate task knowledge representations. The system has been evaluated using a database comprising 200k dialogs drawn from live customer traffic. 1.

Spoken dialogue systems promise efficient and natural access to a large variety of information sources and services from any phone. However, current spoken dialogue systems are deficient in their strategies for preventing, identifying and repairing problems that arise in the conversation. This paper reports results on automatically training a Problematic Dialogue Predictor to predict problematic human-computer dialogues using a corpus of 4692 dialogues collected with the 'How May I Help You' (SM) spoken dialogue system. The Problematic Dialogue Predictor can be immediately applied to the system's decision of whether to transfer the call to a human customer care agent, or be used as a cue to the system's dialogue manager to modify its behavior to repair problems, and even perhaps, to prevent them. We show that a Problematic Dialogue Predictor using automatically-obtainable features from the first two exchanges in the dialogue can predict problematic dialogues 13.2% more accurately than the baseline.

We are interested in spoken language understanding within the domain of automated telecommunication services. Our current methodology involves training statistical language models from large annotated corpora for recognition and understanding. Since the transcribing of large speech corpora is a resource consuming task, we are motivated to exploit speech without transcriptions. In particular, we learn the semantic associations for a task exploiting only phone-based sequences from the output of a task-independent ASR-system. In this paper we present a new multipass algorithm for acquiring salient phone sequences from untranscribed speech corpora and evaluate their utility for the HMIHY task. Compared to our previous strategy, this algorithm is shown to produce improved call-classification results while reducing up to 7-fold the number of salient phone-sequences selected for training.

Current methods for training statistical language models for recognition and understanding require large annotated corpora. The collection, transcription and labeling of such corpora is a major bottleneck for creating new applications and for refinements of existing ones. Thus, it is of great interest to develop methods for automatically learning vocabulary, grammar and semantics from a speech corpus without transcriptions. In this paper we report on an experiment where acoustic morphemes are automatically acquired from the output of a task-independent phone recognizer. The utility of these units is experimentally evaluated for call-type classification in the 'How may I help you?' task. Detected occurrences of the acoustic morphemes in the lattice output provide the basis for the classification of the test sentences. Using lattices, we achieve a reduction of 59% from the false rejection rate using best paths, albeit with a 5% reduction in the correct classification performance from that baseline.

We are interested in adaptive spoken dialog systems for automated services. Peoples' spoken language usage varies over time for a given task, and furthermore varies depending on the state of the dialog. Thus, it is crucial to adapt automatic speech recognition (ASR) language models to these varying conditions. We characterize and quantify these variations based on a database of 30 K user-transactions with AT&T's experimental How May I Help You? spoken dialog system. We describe a novel adaptation algorithm for language models with time and dialog-state varying parameters. Our language adaptation framework allows for recognizing and understanding unconstrained speech at each stage of the dialog, enabling context-switching and error recovery. These models have been used to train state-dependent ASR language models. We have evaluated their performance with respect to word accuracy and perplexity over time and dialog states. We have achieved a reduction of 40% in perplexity and of 8.4% in word error rate over the baseline system, averaged across all dialog states.

In this paper we describe a systematic approach for creating a dialog management system based on a Construct Algebra, a collection of relations and operations on a task representation. These relations and operations are analytical components for building higher level abstractions called dialog motivators. The dialog manager, consisting of a collection of dialog motivators, is entirely built using the Construct Algebra.

We are interested in adaptive spoken dialog systems for automated services. Peoples' spoken language usage varies over time for a fixed task, and furthermore varies depending on the state of the dialog. We characterize and quantify this variation based on a database of 20 K user-transactions with AT&T's experimental 'How May I Help You?' spoken dialog system. We then report on a language adaptation algorithm which was used to train state-dependent ASR language models, experimentally evaluating their improved performance with respect to word accuracy and perplexity.

This paper is concerned with language modeling (LM) for large vocabulary speech recognition in Mandarin Chinese. As the language characteristics of Chinese are quite unique, we investigate some novel techniques in language modeling. We also borrow some of techniques that have been applied to other languages. Experiments have been conducted on the Call Home Mandarin, HUB4, and HUB5 corpora obtained from the Linguistic Data Consortium (LDC). The training set consists of 9.8 hours of spontaneous speech and 100K words in text. The test set consists of 1.6 hours of spontaneous speech and 20K words in text. We have found that our results compare favorably to the results reported in the literature.

A new method for automatically acquiring Fragments for understanding fluent speech is proposed. The goal of this method is to generate a collection of Fragments, each representing a set of syntactically and semantically similar phrases. First, phrases observed frequently in the training set are selected as candidates. Each candidate phrase has three associated probability distributions: of following contexts, of preceding contexts, and of associated semantic actions. The similarity between candidate phrases is measured by applying the Kullback-Leibler distance to these three probability distributions. Candidate phrases that are close in all three distances are clustered into a Fragment. Salient sequences of these Fragments are then automatically acquired, and exploited by a spoken language understanding module to classify calls in AT&T's "How May I Help You?" task. These Fragments allow us to generalize unobserved phrases. For instance, they detected 246 phrases in the test-set that we...

Stochastic language models for speech recognition have traditionally been designed and evaluated in or-der to optimize word accuracy. In this work, we present a novel framework for training stochastic language models by optimizing two different criteria appropri-ate for speech recognition and language understand-ing. First, the language entropy and salience measure are used for learning the relevant spoken language features (phrases). Secondly, a novel algorithm for training stochastic finite state machines is presented which incorporates the acquired phrase structure into a single stochastic language model. Thirdly, we show the benefit of our novel framework with an end-to-end evaluation of a large vocabulary spoken language system for call routing. 2.

We describe a procedure for contextual interpretation of spoken sentences within dialogs. Task structure is represented in a graphical form, enabling the interpreter algorithm to be efficient and task-independent. Recognized spoken input may consist either of a single sentence with utterance-verification scores, or of a word lattice with arc weights. A confidence model is used throughout and all inferences are probability-weighted. The interpretation consists of a probability for each class and for each auxiliary information label needed for task completion. Anaphoric references are permitted. 1. INTRODUCTION We are interested in spoken dialog systems in which the caller responds using fluent natural language to the prompt "How may I help you?" (HMIHY). In previous work we have described the speech recognizer [1], automatic acquisition of salient phrase and grammar fragments, and call-type classification [2,3], the dialog manager [4], and the incorporation of utterance verification [...

We are interested in the problem of understanding fluently spoken language. In particular, we consider people's responses to the open-ended prompt of "How may I help you?". We then further restrict the problem to classifying and automatically routing such a call, based on the meaning of the user's response. Thus, we aim at extracting a relatively small number of semantic actions from the utterances of a very large set of users who are not trained to the system's capabilities and limitations. In this paper, we describe the main components of our speech understanding system: the large vocabulary recognizer and the language understanding module performing the call-type classification. In particular, we propose automatic algorithms for selecting phrases from a training corpus in order to enhance the prediction power of the standard word n-gram. The phrase language models are integrated into stochastic finite state machines which outperform standard word n-gram language models. From the speech recognizer output we recognize and exploit automatically-acquired salient phrase fragments to make a call-type classification. This system is evaluated on a database of 10 K fluently spoken utterances collected from interactions between users and human agents.

Spoken dialog systems interpret a user's request and engage in conversation if the need arises. It is the responsibility of the dialog manager to determine if this need is present and how to proceed. Our spoken dialog system is constructed to sufficiently understand a user's response to the open-ended prompt 'How may I help you?' in order to route a caller to an appropriate destination, with subsequent processing for information retrieval or call completion. In this paper we describe how to structure the relationships among the call types into an inheritance hierarchy. We then describe an algorithm which exploits this hierarchy and the output of a spoken language understanding module to generate a set of semantically consistent inputs.

The paper continues the authors' investigation of machines that adaptively acquire language through interaction with a complex environment. In particular, the present work focuses on the problem of spoken word acquisition, using the authors' proposed principles to motivate a method to govern the emergence of word symbols from the speech signal. The mechanism involves a connectionist network embedded in a feedback control system. The resulting system has two unique characteristics. First, no text is utilized by the device, in contrast to all other speech understanding systems. Second, the vocabulary and grammar is unconstrained, being acquired by the device during the course of performing its task. This is also in contrast to all other systems, in which the salient vocabulary and grammar are preprogrammed. A rudimentary baseline experiment is described, involving 1105 natural language utterances in an automated call routing application scenario.

The goal of this special issue is to present a representative set of current research papers that address the topic of neural networks in speech processing. The application areas addressed include speech analysis, synthesis, recognition and understanding. Due to the large volume of current research in these areas, the authors make the usual disclaimer and apologize for any work that may not be represented in the limited space allocated here. However, the included papers provide a fairly broad overview of each area, as well as citations to the related literature.>

We report on progress in understanding how to build machines which adaptively acquire the language for their task. The generic mechanism in our research has been an information-theoretic connectionist network embedded in a feedback control system. In this paper, we investigate the capability of such a network to learn associations between messages and meaningful responses to them as a task increases in size and complexity. Specifically, we consider how one might reflect task structure in a network architecture in order to provide improved generalization capability in language acquisition. We propose a method for constructing networks from component subnetworks, namely a product network, which provides improved generalization by factoring the associations between words and actions through an intermediate layer of semantic primitives. A two-dimensional product network was evaluated in a 1000-action data retrieval system, the object of which is to answer questions about 20 attributes of the 50 states of the USA. The system was tested by 13 subjects over a two-week period, during which over 1000 natural language dialogues were recorded. The experiment was conducted using typed input with unconstrained vocabulary and syntax. During the course of performing its task, the system acquired over 500 words and retained 92% of what it learned. We provide a description of the system and details on the experimental results.

The authors report on the progress in understanding how to build devices which adaptively acquire the language for their task. The generic device is an information-theoretic connectionist network embedded in a feedback control system. They investigate the capability of the network to learn associations between messages and meaningful responses to them as a task increases in size and complexity. Specifically, the authors consider how one might reflect task structure in the network architecture in order to provide improved generalization capability in language acquisition. They propose a product network, which provides improved generalization by factoring the associations between words and action through semantic primitives. The product network is being evaluated in several experimental systems, including a 1000-action Almanac data retrieval system. They describe these systems and provide details on two preliminary experiments.>

The problem of building a device that acquires language during the course of performing its task, called learning by doing, is considered. Some basic principles and mechanisms upon which such a device might be constructed are described. In particular, a language acquisition mechanism that is based upon the intuition of building associations between messages and appropriate responses to them and a mechanism for human-machine interaction based on control theory methods are investigated. A conversational-mode system that demonstrates and evaluates the proposed principles and mechanisms is described. Experimental results that validate the approach are reported.>

A system that automatically acquires a language model for a particular task from semantic-level information is described. This is in contrast to systems with predefined vocabulary and syntax. The purpose of the system is to map spoken or typed input into a machine action. To accomplish this task a medium-grain neural network is used. An adaptive training procedure is introduced for estimating the connection weights. It has the advantages of rapid, single-pass and order-invariant learning. The resulting weights have information-theoretic significance and do not require gradient search techniques for their estimation. The system was experimentally evaluated on three text-based tasks; a three-class inward-call manager with an acquired vocabulary of over 1600 words, a 15-action subset of the DARPA Resource Manager with an acquired vocabulary of over 700 words, and discrimination between idiomatic phrases meaning yes or no.>

The authors investigate the parallel computation of large-vocabulary speech recognition on a tree-structured parallel processor. Having seriously established results on parallel level-building with continuously variable hidden Markov models they extend this work to the case in which the spoken sentences are constrained by a finite state grammar. The two key ideas are: (1) a pipelined sorting function on the processor array that efficiently transmits a sorted list of the best scores from all processors to the host; and (2) a level-based pruning technique in which paths through the dynamic programming network are pruned only at the ends of words. These ideas are evaluated on the BT-100 processor, a binary-tree parallel processor. A performance model is presented that estimates execution time as a function of algorithm parameters. Real-time speech recognition has been achieved for a data-entry task with a 70-word vocabulary and average branching factor of 23, and for an airline reservation task with a vocabulary of 132 words. The performance model predicts real-time execution of the 991-word DARPA Resource Management Task on a 127-processor machine.>

Describes a parallel computer architecture targeted at signal pattern analysis applications, scalable to configurations capable of TeraFLOP (10/sup 12/ floating point operations per second) throughput. An important attribute of the architecture is its low interconnection overhead, making it well suited to miniaturization using advanced packaging. Preliminary design and thermal tests project a computing density of 300 GigaFLOPS per cubit foot. The architecture is reconfigurable as a tree machine, one or more rings, or a set of linear systolic arrays. Fault tolerance is achieved by embedding these topologies within a four-connected lattice, growing around any faults. A performance model is derived and used to analyze the impact of skewness of the embedded trees on the execution time of parallel recognition algorithms.>

The architecture of the AT&T DSP-3 parallel processor is described. The DSP-3 design is modular and when implemented with 128 processing nodes, provides a maximum throughput of 3.2 GFLOPS (32 bit floating point). The high speed interconnection network (40 Mbytes/sec) contains redundant paths that allow the machine to be configured in a variety of topologies. This flexibility supports efficient operation for a diverse set of signal processing applications and enables topology reconfiguration in support of fault tolerance. Advanced multi-chip 3-D packaging will allow a 800 MFLOP version of the machine to be realized in a multi-processor array volume of 9 cubic inches.

The authors describe a parallel frame-synchronous level-building algorithm, utilizing HMM word-models, for connected-speech recognition on a tree-structured parallel computer. The algorithm is scalable in the sense that the source code and execution time remain essentially the same as vocabulary size increases, so long as the hardware is scaled proportionally. An illustrative sizing and timing analysis of a speaker-independent connected-digit recognizer on the ASPEN tree-machine is described. This algorithm executes in real-time and achieves 98.3% string accuracy on the Texas Instruments digit data base.>

The investigation of parallel computer architectures and parallel algorithms for speech recognition is important. Large vocabulary speech recognition is a computationally intensive problem, which can require orders of magnitude acceleration over a single processor to achieve real-time performance. Also, there is still much algorithm development work to be done, which requires a programmable computer rather than a fixed hardware implementation. This paper describes progress on the ASPEN parallel computer, that is applicable to signal understanding problems in general and large-vocabulary speech recognition in particular. The following will be described in this paper: the hardware architecture; the software architecture; classes of pattern recognition algorithms that are well-suited to the architecture; examples from speech recognition.

The investigation of multiprocessor architectures and parallel algorithms for speech recognition is important. Large vocabulary speech recognition is a computationally intensive problem, which can require orders of magnitude acceleration over uniprocessors to achieve real-time performance. Also, there is still much algorithm development work to be done, which requires a programmable computer rather than a hardware implementation. This paper describes a massively parallel hardware/software architecture that is applicable to accelerating a wide class of large vocabulary speech recognition algorithms. The general principles of applicability supporting this claim will be described. Timing and sizing results obtained by applying these principles to Rabiner's level-building DTW algorithm for connected-word recognition will be given. Finally, a benchmark algorithm is described that demonstrates the programmability and performance of the architecture.