With robotics rapidly advancing, more effective human–robot interaction is increasingly needed to realize the full potential of robots for society. While spoken language must be part of the solution, our ability to provide spoken language interaction capabilities is still very limited. In this article, based on the report of an interdisciplinary workshop convened by the National Science Foundation, we identify key scientific and engineering advances needed to enable effective spoken language interaction with robotics. We make 25 recommendations, involving eight general themes: putting human needs first, better modeling the social and interactive aspects of language, improving robustness, creating new methods for rapid adaptation, better integrating speech and language with other communication modalities, giving speech and language components access to rich representations of the robot’s current knowledge and state, making all components operate in real time, and improving research infrastructure and resources. Research and development that prioritizes these topics will, we believe, provide a solid foundation for the creation of speech-capable robots that are easy and effective for humans to work with.

In this paper, we explore the model combination problem for rescoring Automatic Speech Recognition (ASR) hypotheses. We use minimum Empirical Bayes Risk for the optimization criterion and Deterministic Annealing techniques to search through the non-convex parameter space. Our experiments on the DARPA WSJ task using several different language models showed that our approach consistently outperforms the standard methods of model combination that optimize using 1-best hypothesis error.

Floor control is a scheme used by people to organize speaking turns in multi-party conversations. Identifying the floor control shifts is important for understanding a conversation's structure and would be helpful for more natural human computer interaction systems. Although people tend to use verbal and nonverbal cues for managing floor control shifts, only audio cues, e.g., lexical and prosodic cues, have been used in most previous investigations on speaking turn prediction. In this paper, we present a statistical model to automatically detect floor control shifts using both verbal and nonverbal cues. Our experimental results show that using a combination of verbal and nonverbal cues provides more accurate detection.

Predicting syntactic information in a joint language model (LM) has been shown not only to improve the model at its main task of predicting words, but it also allows this information to be passed to other applications, such as spoken language processing. This raises the question of just how accurate the syntactic information predicted by the LM is. In this paper, we present a joint LM designed not only to scale to large quantities of training data, but also to be able to utilize fine-grain syntactic information, as well as other features, such as morphology and prosody. We evaluate the accuracy of our model at predicting syntactic information on the POS tagging task against state-ofthe-art POS taggers and on perplexity against the ngram model. Index Terms: language modeling, joint language model, partof-speech tagging

We describe and analyze inference strategies for combining outputs from multiple question answering systems each of which was developed independently. Specifically, we address the DARPA-funded GALE information distillation Year 3 task of finding answers to the 5-Wh questions (who, what, when, where, and why) for each given sentence. The approach we take revolves around determining the best system using discriminative learning. In particular, we train support vector machines with a set of novel features that encode systems’ capabilities of returning as many correct answers as possible. We analyze two combination strategies: one combines multiple systems at the granularity of sentences, and the other at the granularity of individual fields. Our experimental results indicate that the proposed features and combination strategies were able to improve the overall performance by 22% to 36% relative to a random selection, 16% to 35% relative to a majority voting scheme, and 15% to 23% relative to the best individual system. Index Terms: Question answering, Systems for spoken language understanding

This paper looks at a parsing-based alternative to word error rate (WER) for optimizing recognition, SParseval, hypothesizing that it may be a better objective for applications such as translation. We find that SParseval is more correlated than WER with human measures of subsequent translation performance, but that optimizing explicitly for SParseval does not give a significant reduction in translation error as measured by automatic methods based on a single translation reference. However, anecdotal examples indicate that SParseval does improve automatic speech recognition (ASR) results, leaving open the possibility that it may be more useful in the future or for other language processing tasks.

In this paper, we investigate bootstrapping part-of-speech (POS) taggers for Mandarin broadcast news (BN) transcripts using co-training, by iteratively retraining two competitive POS taggers from a small set of labeled training data and a large set of unlabeled data. We compare co-training with self-training and our results show that the performance using co-training is significantly better than that from self-training and these semi-supervised learning methods significantly improve tagging accuracy over training only on the small labeled seed corpus. We also investigate a variety of example selection approaches for co-training and find that the computationally expensive, agreement-based selection approach and a more efficient selection approach based on maximizing training utility produce comparable tagging performance from resulting POS taggers. By applying co-training, we are able to build effective POS taggers for Mandarin transcribed speech with the tagging accuracy comparable to that obtained on newswire text.

We present a novel probabilistic model for the hierarchal structure of an image and its regions. We call this model spatial random tree grammars (SRTGs). We develop algorithms for exact computation of likelihoods and MAP estimates and exact EM updates for model-parameter estimation. We collectively call these algorithms the center-surround algorithm. We use the center-surround algorithm to automatically estimate the ML parameters of SRTGs, classify images based on their likelihood and based on the MAP estimate of the associated hierarchal structure. We apply our method to the task of classifying natural images and demonstrate that the addition of hierarchal structure significantly improves upon the performance of a baseline model that lacks such structure.

A grammatical method of combining two kinds of speech repair cues is presented. One cue, prosodic disjuncture, is detected by a decision tree-based ensemble classifier that uses acoustic cues to identify where normal prosody seems to be interrupted (Lickley, 1996). The other cue, syntactic parallelism, codifies the expectation that repairs continue a syntactic category that was left unfinished in the reparandum (Levelt, 1983). The two cues are combined in a Treebank PCFG whose states are split using a few simple tree transformations. Parsing performance on the Switchboard and Fisher corpora suggests that these two cues help to locate speech repairs in a synergistic way.

We present a reranking approach to sentence-like unit (SU) boundary detection, one of the EARS metadata extraction tasks. Techniques for generating relatively small n-best lists with high oracle accuracy are presented. For each candidate, features are derived from a range of information sources, including the output of a number of parsers. Our approach yields significant improvements over the best performing system from the NIST RT-04F community evaluation

Accurate identification of sentence units (SUs) in spontaneous speech has been found to improve the accuracy of speech recognition, as well as downstream applications such as parsing. In recent multimodal investigations, gestur]al features were utilized, in addition to lexical and prosodic cues from the speech channel, for detecting SUs in conversational interactions using a hidden Markov model (HMM) approach. Although this approach is computationally efficient and provides a convenient way to modularize the knowledge sources, it has two drawbacks for our SU task. First, standard HMM training methods maximize the joint probability of observations and hidden events, as opposed to the posterior probability of a hidden event given observations, a criterion more closely related to SU classification error. A second challenge for integrating gestural features is that their absence sanctions neither SU events nor non-events; it is only the co-timing of gestures with the speech channel that should impact our model. To address these problems, a Maximum Entropy (ME) model is used to combine multimodal cues for SU estimation. Experiments carried out on VACE multi-party meetings confirm that the ME modeling approach provides a solid framework for multimodal integration.

This work looks at the impact of automatically predicted commas on part-of-speech (POS) and name tagging of speech recognition transcripts of Mandarin broadcast news. There is a significant gain in both POS and name tagging accuracy due to using automatically predicted commas over sentence boundary prediction alone. One difference between Mandarin and English is that there are two types of commas, and experiments here show that, while they can be reliably distinguished in automatic prediction, the distinction does not give a clear benefit for POS or name tagging.

In October 2003, a group of multidisciplinary researchers convened at the Symposium on Next Generation Automatic Speech Recognition (ASR) to consider new directions in building ASR systems (Lee 2003).Although the workshop's goal of "integrating multidisciplinary sources of knowledge, from acoustics, speech, linguistics, cognitive science, signal processing, human computer interaction, and computer science, into every stage of ASR component and system design" is an important goal, there remains a divide among these communities that can only be addressed through the educational process.The book Introducing Speech and Language Processing by John Coleman represents a bold effort to educate students in speech science about some of the important methods used in speech and natural language processing (NLP).This book represents an important first step for forging effective collaborations with the speech and language processing communities.Coleman states in chapter 1 of his book that "This is a first, basic, elementary and short textbook in speech and natural language processing for beginners with little or no previous experience of computer programming" (page 2).Coleman targets the book at students in a variety of disciplines, including arts, humanities, linguistics, psychology, and speech science, as well as early science and engineering students who want a glimpse into natural language and speech processing.However, since it assumes prior knowledge of basic linguistics, the text is likely to be less accessible to traditional science and engineering students.Coleman's motivation for writing this book is that the currently available textbooks in NLP and speech require knowledge that students from more of a humanities background would not have (e.g., programming, signal processing).The author also astutely points out that there tends to be a divide between the areas of signal processing and computational linguistics, although in recent years with ubiquity of statistical modeling and machine learning techniques in both areas, this divide is becoming much smaller.The author's motivation for this book is excellent: "a refusal to let the old sociological divide between arts and sciences stand in the way of a new wave of spoken language researchers with a foot in both camps" (page 4).The textbook covers a variety of techniques in speech and natural language processing, along with computer programs implementing many of them in either C or Prolog, and it capitalizes on Coleman's insights from courses offered to graduate linguistics students.It comes with a companion CD containing software needed to compile and/or execute the programs in the book, as well as source code for all of the described implementations.The readme file on the CD contains helpful installation notes, while the text describes how to compile and use each of the programs.Chapter 1 contains

Effective human and automatic processing of speech requires recovery of more than just the words. It also involves recovering phenomena such as sentence boundaries, filler words, and disfluencies, referred to as structural metadata. We describe a metadata detection system that combines information from different types of textual knowledge sources with information from a prosodic classifier. We investigate maximum entropy and conditional random field models, as well as the predominant hidden Markov model (HMM) approach, and find that discriminative models generally outperform generative models. We report system performance on both broadcast news and conversational telephone speech tasks, illustrating significant performance differences across tasks and as a function of recognizer performance. The results represent the state of the art, as assessed in the NIST RT-04F evaluation

We develop a new class of hierarchical stochastic image models called spatial random trees (SRTs) which admit polynomial-complexity exact inference algorithms. Our framework of multitree dictionaries is the starting point for this construction. SRTs are stochastic hidden tree models whose leaves are associated with image data. The states at the tree nodes are random variables, and, in addition, the structure of the tree is random and is generated by a probabilistic grammar. We describe an efficient recursive algorithm for obtaining the maximum a posteriori estimate of both the tree structure and the tree states given an image. We also develop an efficient procedure for performing one iteration of the expectation-maximization algorithm and use it to estimate the model parameters from a set of training images. We address other inference problems arising in applications such as maximization of posterior marginals and hypothesis testing. Our models and algorithms are illustrated through several image classification and segmentation experiments, ranging from the segmentation of synthetic images to the classification of natural photographs and the segmentation of scanned documents. In each case, we show that our method substantially improves accuracy over a variety of existing methods.

Sentence boundary detection in speech is important for enriching speech recognition output, making it easier for humans to read and downstream modules to process. In previous work, we have developed hidden Markov model (HMM) and maximum entropy (Maxent) classifiers that integrate textual and prosodic knowledge sources for detecting sentence boundaries. In this paper, we evaluate the use of a conditional random field (CRF) for this task and relate results with this model to our prior work. We evaluate across two corpora (conversational telephone speech and broadcast news speech) on both human transcriptions and speech recognition output. In general, our CRF model yields a lower error rate than the HMM and Maxent models on the NIST sentence boundary detection task in speech, although it is interesting to note that the best results are achieved by three-way voting among the classifiers. This probably occurs because each model has different strengths and weaknesses for modeling the knowledge sources.

Both human and automatic processing of speech require recognition of more than just words. In this paper we provide a brief overview of research on structural metadata extraction in the DARPA EARS rich transcription program. Tasks include detection of sentence boundaries, filler words, and disfluencies. Modeling approaches combine lexical, prosodic, and syntactic information, using various modeling techniques for knowledge source integration. The performance of these methods is evaluated by task, by data source (broadcast news versus spontaneous telephone conversations) and by whether transcriptions come from humans or from an (errorful) automatic speech recognizer. A representative sample of results shows that combining multiple knowledge sources (words, prosody, syntactic information) is helpful, that prosody is more helpful for news speech than for conversational speech, that word errors significantly impact performance, and that discriminative models generally provide benefit over maximum likelihood models. Important remaining issues, both technical and programmatic, are also discussed.

Automatic detection of disfluencies in spoken language is important for making speech recognition output more readable, and for aiding downstream language processing modules. We compare a generative hidden Markov model (HMM)-based approach and two conditional models — a maximum entropy (Maxent) model and a conditional random field (CRF) — for detecting disfluencies in speech. The conditional modeling approaches provide a more principled way to model correlated features. In particular, the CRF approach directly detects the reparandum regions, and thus avoids the use of ad-hoc heuristic rules. We evaluate performance of these three models across two different corpora (conversational speech and broadcast news) and for two types of transcriptions (human transcriptions and recognition output). Overall we find that that the conditional modeling approaches (Maxent and CRF) provide benefit over the HMM approach. Effects of speaking style, word recognition errors, and future directions are also discussed. 1.

Structured language models have recently been shown to give significant improvements in large-vocabulary recognition relative to traditional word N-gram models, but typically imply a heavy computational burden and have not been applied to large training sets or complex recognition systems. Previously, we developed a linguistically motivated and computationally efficient almost-parsing language model, using a data structure derived from constraint dependency grammar parsing, that tightly integrates knowledge of words, lexical features, and syntactic constraints. We show that such a model can be used effectively and efficiently in all stages of a complex, multi-pass conversational telephone speech recognition system. Compared to a state-of-the-art 4-gram interpolated word- and class-based language model, we obtained a 6.2% relative word error reduction (a 1.6% absolute reduction) on a recent NIST evaluation set.

In this paper, we adopt a direct modeling approach to utilize conversational gesture cues in detecting sentence boundaries, called SUs, in video taped conversations. We treat the detection of SUs as a classification task such that for each inter-word boundary, the classifier decides whether there is an SU boundary or not. In addition to gesture cues, we also utilize prosody and lexical knowledge sources. In this first investigation, we find that gesture features complement the prosodic and lexical knowledge sources for this task. By using all of the knowledge sources, the model is able to achieve the lowest overall SU detection error rate.

We investigate machine learning techniques for coping with highly skewed class distributions in two spontaneous speech processing tasks. Both tasks, sentence boundary and disfluency detection, provide important structural information for downstream language processing modules. We examine the effect of data set size, task, sampling method (no sampling, downsampling, oversampling, and ensemble sampling), and learning method (bagging, ensemble bagging, and boosting) for a decision tree prosody model. Results show that (1) bagging benefits both tasks, but to different degrees, (2) the benefit from ensemble bagging decreases as data size increases, and (3) boosting can outperform bagging under certain conditions.

Both human and automatic processing of speech require recognizing more than just the words. We describe a state-of-the-art system for automatic detection of “metadata” (information beyond the words) in both broadcast news and spontaneous telephone conversations, developed as part of the DARPA EARS Rich Transcription program. System tasks include sentence boundary detection, filler word detection, and detection/correction of disfluencies. To achieve best performance, we combine information from different types of language models (based on words, part-of-speech classes, and automatically induced classes) with information from a prosodic classifier. The prosodic classifier employs bagging and ensemble approaches to better estimate posterior probabilities. We use confusion networks to improve robustness to speech recognition errors. Most recently, we have investigated a maximum entropy approach for the sentence boundary detection task, yielding a gain over our standard HMM approach. We report results for these techniques on the official NIST Rich Transcription metadata tasks.

A new class of multiscale multidimensional stochastic processes called spatial random trees is introduced. The model is based on multiscale stochastic trees with stochastic structure as well as stochastic states. Procedures are developed for exact likelihood calculation, MAP estimation of the process, and estimation of the parameters of the process. The new framework is illustrated through a simple binary image classification problem.

An almost-parsing language model has been developed (Wang and Harper 2002) that provides a framework for tightly integrating multiple knowledge sources. Lexical features and syntactic constraints are integrated into a uniform linguistic structure (called a SuperARV) that is associated with words in the lexicon. The SuperARV language model has been found able to reduce perplexity and word error rate (WER) compared to trigram, part-of-speech-based, and parser-based language models on the DARPA Wall Street Journal (WSJ) CSR task. In this paper we further investigate the robustness of the language model to possibly inconsistent and flawed training data, as well as its ability to scale up to sophisticated LVCSR tasks by comparing performance on the DARPA WSJ and Hub4 (Broadcast News) CSR tasks.

A new class of multiscale multidimensional stochastic processes called spatial random trees was recently introduced in J. Pollak et al. (2003). The model is based on multiscale stochastic trees with stochastic structure as well as stochastic states. In this work, we describe a method for estimating the parameters of the process.

A new almost-parsing language model incorporating multiple knowledge sources that is based upon the concept of constraint Dependency Grammars is presented in this paper. Lexical features and syntactic constraints are tightly integrated into a uniform linguistic structure called a SuperARV that is associated with a word in the lexicon. The SuperARV language model reduces perplexity and word error rate compared to trigram, part-of-speech-based, and parser-based language models. The relative contributions of the various knowledge sources to the strength of our model are also investigated by using constraint relaxation at the level of the knowledge sources. We have found that although each knowledge source contributes to language model quality, lexical features are an outstanding contributor when they are tightly integrated with word identity and syntactic constraints. Our investigation also suggests possible reasons for the reported poor performance of several probabilistic dependency grammar models in the literature.

In this paper, we compare the efficacy of a variety of language models (LMs) for rescoring word graphs and N-best lists generated by a large vocabulary continuous speech recognizer. These LMs differ based on the level of knowledge used (word, lexical features, syntax) and the type of integration of that knowledge (tight or loose). The trigram LM incorporates word level information; our Part-of-Speech (POS) LM uses word and lexical class information in a tightly coupled way; our new SuperARV LM tightly integrates word, a richer set of lexical features than POS, and syntactic dependency information; and the Parser LM integrates some limited word information, POS, and syntactic information. We also investigate LMs created using a linear interpolation of LM pairs. When comparing each LM on the task of rescoring word graphs or N-best lists for the Wall Street Journal (WSJ) 5k- and 20k- vocabulary test sets, the SuperARV LM always achieves the greatest reduction in word error rate (WER) and the greatest increase in sentence accuracy (SAC). On the 5k test sets, the SuperARV LM obtains more than a 10% relative reduction in WER compared to the trigram LM, and on the 20k test set more than 2%. Additionally, the SuperARV LM performs comparably to or better than the interpolated LMs. Hence, we conclude that the tight coupling of knowledge from all three levels is an effective method of constructing high quality LMs.

Speech and gesture are two primary modes used in natural human communication; hence, they are important inputs for a multimodal interface to process. One of the challenges for multimodal interfaces is to accurately recognize the words in spontaneous speech. This is partly due to the presence of speech repairs, which seriously degrade the accuracy of current speech recognition systems. Based on the assumption that speech and gesture arise from the same thought process, we would expect to find patterns of gesture that co-occur with speech repairs that can be exploited by a multimodal processing system to more effectively process spontaneous speech. To evaluate this hypothesis, we have conducted a measurement study of gesture and speech repair data extracted from videotapes of natural dialogs. Although we have found that gestures do not always co-occur with speech repairs, we observed that modification gesture patterns have a high correlation with content replacement speech repairs, but rarely occur with content repetitions. These results suggest that gesture patterns can help us to classify different types of speech repairs in order to correct them more accurately.

Parkinson’s disease (PD) belongs to a class of neurodegenerative diseases that affect the patient’s speech, motor, and cognitive capabilities. All three deÞcits affect the multimodal communication channels of speech and gesture. We presenta study on the changes in speech pause patterns and gesture holds before and after treatment. We present the results of a pilot study of two Idiopathic PD patients who have undergone Lee Silverman Voice Treatment (LSVT). We show that therewas a consistentchangein the location of pauses with respect to semantic sentential utterance units. After treatment, the number and duration of pauses within sentential units decreased while the inter sentential pauses increased. This indicates reduction in hesitation and increase in speech phrasing. We also found a decrease in the number of sentence repairs and the time spent in repairs. For gesture, we found that non-rest holds intersecting with within-sentence pauses appears to decline after treatment, as does the ratio of rest holds during speechagainst rest holds between sentences. While the work is preliminary, these patterns suggest that multimodal discourse characteristics might provide accessto the underlying cognitive state under the load of narrative discourse.

Human multimodal communicative behaviorsform a tightly integrated whole. By matching up gestural features with a carefully time-tagged transcription of the speech, we can observe how gesture features and discourse unit transitions cohere. Space usage SU is a key gestural component. We summarize the theory of SU. In our experimentswhere subjects make action plans around a terrain map, such SU become key organizational loci around which the discourse may be built. Our vision-based approach extracts ‘SU histograms’ from stereo video describing the locus of motion of a speaker’s dominant hand. An ‘fuzzy correlation’ of these histograms yields a correlation space in which similar SU is clustered. By locating the cluster transitions we can locate topical shiftsinthediscourse. Weshowresultsbycomparingthetransitions extracted from a sentential codingwith a psycholinguisticsemantic coding. We do the same with a uniform distributed time units and demonstratethe ability to recoverdiscoursetransitions.

Parkinson's disease (PD) belongs to a class of neurodegenerative diseases that affect both the patient's speech and motor capabilities. To date, PD diagnosis and the determination of disease progress and treatment efficacy is based entirely on the subjective observation of a trained physician. We present the results of a pilot study of two Idiopathic PD patients who have undergone Lee Silverman Voice Treatment (LSVT). It has been observed subjectively that gestural performance of patients improve in tandem with speech improvements after LSVT. It is hypothesized that these improvements are taking place at a neurological level. Measurements of speech and gesture suggest that LSVT improves the quality of both gesticulation and speech.

This paper describes an extension to the hidden Markov model for part-of-speech tagging using second-order approximations for both contextual and lexical probabilities. This model increases the accuracy of the tagger to state of the art levels. These approximations make use of more contextual information than standard statistical systems. New methods of smoothing the estimated probabilities are also introduced to address the sparse data problem.

We describe a prototype spoken language system that loosely integrates a speech recognition component based on hidden Markov models with a constraint dependency grammar (CDG) parser using a word graph to pass sentence candidates between the two modules. This loosely coupled system was able to improve the sentence selection accuracy and concept accuracy over the level achieved by the acoustic module with a stochastic grammar. Timing profiles suggest that a tighter coupling of the modules could reduce parsing times of the system, as could the development of better acoustic models and tighter parsing constraints for conjunctions.

Tone classification is important for Thai speech recognition because tone affects the lexical identification of words. An analysis-by-synthesis algorithm for classifying Thai tones in syllable-segmented speech is presented that uses an extension to Fujisaki's (1983) model for tone languages that incorporates tonal assimilation and declination. The classifier correctly identifies all of the tones in 89.1% of the test utterances.

One of the most challenging problems in engineering-and signal processing-education is providing realistic and meaningful design experience. In the Engineering Projects in Community Service (EPICS) program, teams of engineering undergraduates earn academic credit for multi-year projects that solve technology-based problems for community organizations. Key features of EPICS include the long-term nature of the projects; emphasis on "real-world" start-to-finish design; the learning experience embodied in solving ambitious engineering problems; vertical, multidisciplinary teams; development of teamwork and communication skills; and the use of engineering to help the community. We describe the EPICS program and highlight four EPICS signal processing projects: a real-time system to measure speaking rate for Purdue's speech clinic; voice-controlled interactive software to encourage speech in developmentally delayed children; a microphone array hearing aid; and a virtual museum tour and interactive Web-based history games for the Tippecanoe County Historical Association.

The research presented here focuses on implementation and efficiency issues associated with the use of word graphs for interfacing acoustic speech recognition systems with natural language processing systems. The effectiveness of various pruning methods for graph construction is examined, as well as techniques for word graph compression. In addition, the word graph representation is compared to another predominant interface method, the N-best sentence list. 1. INTRODUCTION An important research topic in recent years has been the integration of speech recognition systems with language models [2, 8]. Many systems integrate stochastic language models directly into the speech recognizer. However, a structure in which a front-end acoustic recognizer is interfaced to a separate language processing module allows use of more sophisticated parsing techniques and additional semantic and contextual information to aid in speech understanding. The choice of data representations used to accomplish ...

Hybrids that use a neural network to estimate the output probability for a hidden Markov model (HMM) word recognizer have been competitive with traditional HMM recognizers when both use monophone context. While traditional HMM recognizers can easily utilize more context (e.g., triphones) to achieve better results, the size of the task has made it impractical to use phonetic context directly in the neural network front end of a hybrid. In this paper, we suggest a simple method to incorporate more context by modeling the phone distributions obtained from the neural network. This allows the HMM to easily handle stochastic pronunciations as well as errors from the neural network phone recognizer. The re-estimation equations are derived for the new model. Results for the Resource Management task illustrate that SOHMM increases recognition accuracy for the cases of no grammar, unigram grammar, and word pair grammar.

This paper describes an extension to the constraint satisfaction problem (CSP) called MUSE CSP (MUltiply SEgmented Constraint Satisfaction Problem). This extension is especially useful for those problems which segment into multiple sets of partially shared variables. Such problems arise naturally in signal processing applications including computer vision, speech processing, and handwriting recognition. For these applications, it is often difficult to segment the data in only one way given the low-level information utilized by the segmentation algorithms. MUSE CSP can be used to compactly represent several similar instances of the constraint satisfaction problem. If multiple instances of a CSP have some common variables which have the same domains and constraints, then they can be combined into a single instance of a MUSE CSP, reducing the work required to apply the constraints. We introduce the concepts of MUSE node consistency, MUSE arc consistency, and MUSE path consistency. We then demonstrate how MUSE CSP can be used to compactly represent lexically ambiguous sentences and the multiple sentence hypotheses that are often generated by speech recognition algorithms so that grammar constraints can be used to provide parses for all syntactically correct sentences. Algorithms for MUSE arc and path consistency are provided. Finally, we discuss how to create a MUSE CSP from a set of CSPs which are labeled to indicate when the same variable is shared by more than a single CSP.

We show that many of the errors in a context-dependent phone recognition system are due to poor segmentation. We then suggest a method to incorporate explicit segmentation information directly into the HMM paradigm. The utility of explicit segmentation information is illustrated with experiments involving five types of segmentation information and three methods of smoothing.

Tone classification is a crucial component of any automatic speech recognition system for tone languages. It is imperative that tonal information be incorporated into the word hypothesization process because patterns of pitch (or tones) contribute to the lexical identification of the individual words. In this paper, we present a novel algorithm for automatically classifying Thai tones in connected speech using an analysis-synthesis method based on an extension to Fujisaki's model. We have successfully incorporated into the model two major factors affecting the phonetic realization of tones in connected speech: tonal coarticulation and declination. Also addressed is an F/sub 0/ normalization procedure for achieving speaker-independence. In our preliminary experiment, we were able to achieve 89.1% classification accuracy.

In this paper, we describe a system which is capable of utilizing a variety of knowledge sources to select the most appropriate parse for a spoken sentence. These knowledge sources include syntax, semantics, and contextual information. We discuss one way to utilize contextual information when determining the parse for a sentence. At its simplest level, the system can be thought of as a general-purpose query answering system for multiple topical databases. The user's input would be processed by the language processor which interfaces to the databases with the goal of interacting with the correct database in order to provide a reasonable answer to the user's spoken request. Initially, it analyzes a word graph of sentence hypotheses provided by a speech recognizer using general syntactic and semantic rules. Then, if the utterance is still ambiguous, it utilizes context-specific constraints to further refine the analysis. This brings us closer to developing a more general purpose interface for multiple databases.

Introduces a new recursion that reduces the complexity of training a semi-Markov model with continuous output distributions. It is shown that the cost of training is proportional to M/sup 2/+D, compared to M/sup 2/D with the standard recursion, where M is the observation vector length and D is the maximum allowed duration.>

We show that HMM word recognition using Deller and Snider's ( see ibid., vol.1, no.4, 1993) "any path" procedure makes an assumption of independence that is not made by either the forward or Viterbi algorithms. We also point out that additional savings in execution time can be achieved by precomputation.>

We have extended Maruyama's [5, 6, 7] constraint dependency grammar (CDG) to process a lattice or graph of sentence hypotheses instead of separate text strings. A post-processor to a speech recognizer producing N-best hypotheses generates the word graph representation, which is then augmented with information required for parsing. We will summarize the CDG parsing algorithm and then describe how the algorithm is extended to process a word graph on a single processor machine.

In this paper, we represent singular definite noun phrases as functions in logical form.This representation is designed to model the behaviors of both anaphoric and non-anaphoric, distributive definites.It is also designed to obey the computational constraints suggested in Harper [Har88].Our initial representation of a definite places an upper bound on its behavior given its structure and location in a sentence.Later, when ambiguity is resolved, the precise behavior of the definite is pinpointed.

Tense, temporal adverbs, and temporal connectives provide information about when events described in English sentences occur. To extract this temporal information from a sentence, it must be parsed into a semantic representation which captures the meaning of tense, temporal adverbs, and temporal connectives. Representations were developed for the basic tenses, some temporal adverbs, as well as some of the temporal connectives. Five criteria were suggested for judging these representations, and based on these criteria the representations were judged.