Speech understanding of nonnative language learners' speech is a challenging problem. In this paper, we investigate the use of dialogue context cues to help improve concept error rate (CER) of nonnative speech in a language learning system. Given that the student's task is known, we show that incorporating the game scores to help select the best hypothesis improves the CER. We also introduce a novel N-best fusion method to create a single final hypothesis on the meaning level. The experimental results show that the fusion methods can further improve the CER.

In this paper, we present a framework which harvests grassroots-generated data from the Web (e.g., reviews, blogs), extracts latent information from these data, and provides a multimodal interface for review browsing and inquiring. A prescription-drug domain system is implemented under this framework. Patient-provided drug reviews were collected from various health-related forums, from which significant side effects correlated to each drug type were identified with association algorithms. A multimodal web-based spoken dialogue system was implemented to allow users to inquire about drugs and correlated side effects as well as browsing the reviews obtained from the Web. We report evaluation results on speech recognition, parse coverage and system response.

We present a framework for learning a pronunciation lexicon for an Automatic Speech Recognition (ASR) system from multiple utterances of the same training words, where the lexical identities of the words are unknown. Instead of only trying to learn pronunciations for known words we go one step further and try to learn both spelling and pronunciation in a joint optimization. Decoding based on linguistically motivated hybrid subword units generates the joint lexical search space, which is reduced to the most appropriate lexical entries based on a set of simple pruning techniques. A cascade of letter and acoustic pruning, followed by re-scoring N-best hypotheses with discriminative decoder statistics resulted in optimal lexical entries in terms of both spelling and pronunciation. Evaluating the framework on English isolated word recognition, we achieve reductions of 7.7% absolute on word error rate and 20.9% absolute on character error rate over baselines that use no pruning.

Typically data collection, transcription, language model generation, and deployment are separate phases of creating a spoken language interface. An unfortunate consequence of this is that the recognizer usually remains a static element of systems often deployed in dynamic environments. By providing an API for human intelligence, Amazon Mechanical Turk changes the way system developers can construct spoken language systems. In this work, we describe an architecture that automates and connects these four phases, effectively allowing the developer to grow a spoken language interface. In particular, we show that a human-in-the-loop programming paradigm, in which workers transcribe utterances behind the scenes, can alleviate the need for expert guidance in language model construction. We demonstrate the utility of these organic language models in a voice-search interface for photographs.

Recently we began using Amazon Mechanical Turk (AMT), an Internet marketplace, to deploy our spoken dialogue systems to large audiences for user testing and data collection purposes. This crowdsourcing method of collecting data contrasts with the time- and labor- intensive developer annotation methods. In this paper, we compare these data in various combinations with traditionally-collected corpora for training our speech recognizer's language model. Our results show that AMT text queries are effective for initial language model training for spoken dialogue systems, and that crowd-sourced speech collection within the context of a spoken dialogue framework provides significant improvement.

In this paper, we introduce a new envisioned application for speech which allows users to enter restaurant reviews orally via their mobile device, and, at a later time, update a shared and growing database of consumer-provided information about restaurants. During the intervening period, a speech recognition and NLP based system has analyzed their audio recording both to extract key descriptive phrases and to compute sentiment ratings based on the evidence provided in the audio clip. We report here on our preliminary work moving towards this goal. Our experiments demonstrate that multi-aspect sentiment ranking works surprisingly well on speech output, even in the presence of recognition errors. We also present initial experiments on integrated sentence boundary detection and key phrase extraction from recognition output.

This paper presents a method for extracting both syntactic and semantic tags. An extended CFG parser works in conjunction with an HMM model, which handles unknown words and partially known words, to yield a complete syntactic and semantic interpretation of the utterance. Four experiments and applications were performed using the paradigm to show the usefulness of the approach in processing spoken sentences.

Native speakers of non-tonal languages, such as American English, frequently have difficulty accurately producing the tones of Mandarin Chinese. This paper describes a corpus of Mandarin Chinese spoken by non-native speakers and annotated for tone quality using a simple good/bad system. We examine interrater correlation of the annotations and highlight the differences in feature distribution between native, good non-native, and bad non-native tone productions. We find that the features of tones judged by a simple majority to be bad are significantly different from features from tones judged to be good, and tones produced by native speakers. Index Terms: computer aided language learning, tone evaluation, Mandarin, Chinese

In this research, an iterative and unsupervised Turbo-style algorithm is presented and implemented for the task of automatic lexical acquisition. The algorithm makes use of spoken examples of both spellings and words and fuses information from letter and subword recognizers to boost the overall lexical learning performance. The algorithm is tested on a challenging lexicon of restaurant and street names and evaluated in terms of spelling accuracy and letter error rate. Absolute improvements of 7.2% and 3% (15.5% relative improvement) are obtained in the spelling accuracy and the letter error rate respectively following only 2 iterations of the algorithm.

While a wide variety of grammatical mistakes may be observed in the speech of non-native speakers, the types and frequencies of these mistakes are not random. Certain parts of speech, for example, have been shown to be especially problematic for Japanese learners of English [1]. Modeling these errors can potentially enhance the performance of computer-assisted language learning systems. This paper presents an automatic method to estimate an error model from a non-native English corpus, focusing on articles and prepositions. A fine-grained analysis is achieved by conditioning the errors on appropriate words in the context.

Most automatic speech recognizers use a dictionary that maps words to one or more canonical pronunciations. Such entries are typically hand-written by lexical experts. In this research, we investigate a new approach for automatically generating lexical pronunciations using a linguistically motivated subword model, and refining the pronunciations with spoken examples. The approach is evaluated on an isolated word recognition task with a 2 k lexicon of restaurant and street names. A letter-to-sound model is first used to generate seed baseforms for the lexicon. Then spoken utterances of words in the lexicon are presented to a subword recognizer and the top hypotheses are used to update the lexical base-forms. The spelling of each word is also used to constrain the subword search space and generate spelling-constrained baseforms. The results obtained are quite encouraging and indicate that our approach can be successfully used to learn valid pronunciations of new words.

In this paper, we use subword modeling to learn the pronunciations and spellings of new words. The subwords are generated with a context-free grammar, and are intermediate units between phonemes and syllables. We first evaluate the effectiveness of the subword model in automatically generating the spelling and pronunciation of new words. Then the subword model is embedded in a multi-stage recognizer which consists of word, subword, and letter recognizers. In a preliminary set of experiments, the hybrid system outperforms a large-vocabulary isolated word recognizer. The subword model is also used to improve the performance of the letter recognizer by generating a spelling cohort which is used to train a small letter n-gram. The small letter n-gram has a reduced perplexity compared to a much larger n-gram, and can be used by the letter recognizer for the spoken spelling mode. This could translate to an improved letter error rate in future letter recognition experiments. Index Terms: subword modeling, new word acquisition

Fill-in-the-blank questions, or cloze items, are commonly used in language learning applications. The benefits of personalized items, tailored to the user’s interest and proficiency, have motivated research on automatic generation of cloze items. This paper is concerned with generating cloze items for prepositions, whose usage often poses problems for non-native speakers of English. The quality of a cloze item depends on the choice of distractors. We propose two methods, based on collocations and on non-native English corpora, to generate distractors for prepositions. Both methods are found to be more successful in attracting users than a baseline that relies only on word frequency, a common criterion in past research. Index Terms: computer-assisted language learning, natural language generation

We have recently been conducting research on developing spoken dialogue systems to provide conversational practice for a learner of a foreign language. One of the most critical aspects of such a system is speech recognition errors, since they often take the dialogue thread down a wrong turn that is very confusing to the student and may be irrecoverable. In this paper we report on a machine learning technique to assist the process of selection from a list of N-best candidates based on a high-level description of the semantics of the preceding dialogue. In a user simulation experiment, we show that a significant reduction in sentence error rate can be achieved, from 29.2 % to 23.6%. We have not yet verified that our techniques hold for real user data. Index Terms: dialogue modeling, user simulation, confidence scoring, machine learning

We describe work towards developing a scalable and portable framework for enabling map-based multimodal dialogue interaction over the web. Working in the context of a restaurant-guide system, we show how large information databases harvested from the web can be accommodated in our speech recognizer, parser, and web-based GUI. We compare two dynamic language modeling techniques, which calculate context-dependent weights for the large sets of proper nouns associated with geographical entities such as restaurants and streets. We show that the more fine-grained approach results in a 7.8% reduction in concept error rate. Index Terms: multimodal dialogue system, language modeling, restaurants, maps, world wide web

A computer conversational system can potentially help a foreign-language student improve his/her fluency through practice dialogues. One of its potential roles could be to correct ungrammatical sentences. This paper 1 describes our research on a sentence-level, generation-based approach to grammar correction: first, a word lattice of candidate corrections is generated from an ill-formed input. A traditional n-gram language model is used to produce a small set of N-best candidates, which are then reranked by parsing using a stochastic context-free grammar. We evaluate this approach in a flight domain with simulated ill-formed sentences. We discuss its potential applications in a few related tasks. Index Terms: computer-assisted language learning, dialogue systems, natural language generation.

Portability is an important issue to the viability of a domainspecific translation approach. This paper describes an English to Chinese translation system for flight-domain queries, utilizing an interlingua translation framework that has been successfully applied in the weather domain. Portability of various components is tested, and new technologies to handle parse ambiguities and illformed inputs are developed to enhance the translation framework. Evaluation of translation quality is conducted manually on a set of 432 unseen flight-domain utterances, which are translated into Chinese using a formal method and a new robust back-off method in tandem. We achieved 96.7 % sentence accuracy with a rejection rate of 7.6 % on manual transcripts, and 89.1 % accuracy with an 8.6 % rejection rate on speech input. A game for language learning using the translation capability is currently under development. Index Terms: speech translation, domain portability, natural language understanding, natural language generation.

We explore several language modeling strategies for increasing the recognition accuracy among large sets of proper nouns in a map- based multimodal dialogue system which provides restaurant information. In particular, we evaluate several mechanisms for exploiting dialogue context, the two most promising of which involve a semi- static metropolitan-region based large set of proper nouns competing with a smaller, in-focus subset. We show that these techniques decrease word, concept, and proper noun error rates under several training conditions. We also present a technique to generalize sparse training data through derived templates to improve language model robustness.

We present context-sensitive dynamic classes ‐ a novel mechanism for integrating contextual information from spoken dialogue into a class n-gram language model. We exploit the dialogue system’s information state to populate dynamic classes, thus percolating contextual constraints to the recognizer’s language model in real time. We describe a technique for training a language model incorporating context-sensitive dynamic classes which considerably reduces word error rate under several conditions. Significantly, our technique does not partition the language model based on potentially artificial dialogue state distinctions; rather, it accommodates both strong and weak expectations via dynamic manipulation of a single model.

This paper addresses the issue of large-vocabulary recognition in a specific word class. We propose a two-pass strategy in which only major cities are explicitly represented in the first stage lexicon. An unknown word model encoded as a phone loop is used to detect OOV city names (referred to as rare city names). After which SpeM, a tool that can extract words and word-initial cohorts from phone graphs on the basis of a large fallback lexicon, provides an N-best list of promising city names on the basis of the phone sequences generated in the first stage. This N-best list is then inserted into the second stage lexicon for a subsequent recognition pass. Experiments were conducted on a set of spontaneous telephone-quality utterances each containing one rare city name. We tested the size of the N-best list and three types of language models (LMs). The experiments showed that SpeM was able to include nearly 85% of the correct city names into an N-best list of 3000 city names when a unigram LM, which also boosted the unigram scores of a city name in a given state, was used.

In this paper, we address the issue of generating language model training data during the initial stages of dialogue system development. The process begins with a large set of sentence templates, automatically adapted from other application domains. We propose two methods to filter the raw data set to achieve a desired probability distribution of the semantic content, both on the sentence level and on the class level. The first method utilizes user simulation technology, which obtains the probability model via an interplay between a probabilistic user model and the dialogue system. The second method synthesizes novel dialogue interactions by modeling after a small set of dialogues produced by the developers during the course of system refinement. We evaluated our methodology by speech recognition performance on a set of 520 unseen utterances from naive users interacting with a restaurant domain dialogue system. 1.

This paper describes several experiments aimed at the long term goal of enabling a spoken conversational system to automatically improve its pronunciation lexicon over time through direct interactions with end users and from available Web sources. We selected a set of 200 rare words from the OGI corpus of spoken names, and performed several experiments combining spelling and pronunciation information to hypothesize phonemic baseforms for these words. We evaluated the quality of the resulting baseforms through a series of recognition experiments, using the 200 words in an isolated word recognition task. We also report here on a modification to our letter-to-sound system, utilizing a letter-phoneme -gram language model, either alone or in combination with our original “column-bigram” model, for additional linguistic constraint and robustness. Our experiments confirm our expectation that acoustic information drawn from spoken examples of the words can greatly improve the quality of the baseforms, as measured by the recognition error rate. Our ultimate goal is to allow a spoken dialogue system to automatically expand and improve its baseforms over time as users introduce new words or supply spoken pronunciations of existing words.

We propose an induction algorithm to semi-automate grammar authoring in an interlingua-based machine translation framework. This algorithm uses a pre-existing one-way translation system from some other language to the target language as prior information to infer a grammar for the target language. We demonstrate the system’s effectiveness by automatically inducing a Chinese grammar for a weather domain from its English counterpart, and showing that it can produce high-quality translation from Chinese back to English.

Mixed-initiative spoken dialogue systems today generally allow users to query with a fixed vocabulary and grammar that is determined prior to run-time. This paper presents a spoken dialogue interface enhanced with a dynamic vocabulary capability. One or more word classes can be made dynamic in the speech recognizer and natural language (NL) grammar so that a context-specific vocabulary subset can be incorporated on-the-fly as the context of the dialogue changes, at each dialogue turn. Described is a restaurant information domain which continually updates the restaurant name class, given the dialogue context. We examine progress made to the speech recognizer, natural language parser and dialogue manager in order to support the dynamic vocabulary capability, and present preliminary experimental results conducted from simulated dialogues.

We present research towards developing a pronunciation dictionary that features sensitivity to learners’ native phonology, specifically designed for Korean learners of English-as-a-Foreign-Language (EFL). We envision a future system that can record and process learners’ imitationof thedictionary pronunciation and instantly provide segmental and prosodic feedback on accent. Towards this goal, we have designed and collected a speech corpus to address the phonological and prosodic issues of Korean EFL learners. We leverage the SUMMIT speech recognizer’s ability to model phonological rules to automatically identify non-native phonological phenomena. These phonological rules were carefully constructed to account for the influence of learners’ native language (Korean) on the target language (English). Feedback messages are provided to the learner to point out the non-native phonological variations detected by the speech recognizer in order to help them improve segmental pronunciation. Instructions are also given to the user on the prosodic aspects of the pronunciation, which are based on detected duration and cues. We evaluated the effectiveness of the feedback mechanism by rating 222 English utterances from six native Korean subjects, before and after receiving native-language dependent feedback messages. Human raters judged 61% of the utterances as improved after feedback.

introduction Introduction to mobile and adaptive conversational interfaces Share on Authors: Sharon Oviatt Oregon Health & Science University, Beaverton, OR Oregon Health & Science University, Beaverton, ORView Profile , Stephanie Seneff Massachusetts Institute of Technology, Cambridge, MA Massachusetts Institute of Technology, Cambridge, MAView Profile Authors Info & Claims ACM Transactions on Computer-Human InteractionVolume 11Issue 3September 2004 pp 237–240https://doi.org/10.1145/1017494.1017495Online:01 September 2004Publication History 3citation1,848DownloadsMetricsTotal Citations3Total Downloads1,848Last 12 Months10Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

Thesis: S.M., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2002

Through efforts over the past fifteen years, we have acquired a great deal of experience in designing spoken dialogue systems that provide access to large corpora of data in a variety of different knowledge domains, such as flights, hotels, restaurants, weather, etc. In our recent research, we have begun to shift our focus towards developing tools that enable the rapid development of new applications. This paper addresses a novel approach that drives system design from the on-line knowledge resource. We were motivated by a desire to minimize the need for a pre-determined dialogue flow. In our approach, decisions on dialogue flow are made dynamically based on analyses of data, either prior to user interaction or during the dialogue itself. Automated methods, used to organize numeric and symbolic data, can be applied at every turn, as user constraints are being specified. This helps the user mine through large data sets to a few choices by allowing the system to synthesize intelligent summaries of the data, created on-the-fly at every turn. Moreover automatic methods are ultimately more robust against the frequent changes to on-line content. Simulations generating hundreds of dialogues have produced log files that allow us to assess and improve system behavior, including system responses and interactions with the dialogue flow. Together, these techniques are aimed towards the goal of instantiating new domains with little or no input from a human developer. 1.

This paper describes recent advances we have made towards the goal of empowering end users to automatically expand the knowledge base of a dialogue system through spoken interaction, in order to personalize it to their individual needs. We describe techniques used to incrementally reconfigure a preloaded trained natural language grammar, as well as the lexicon and language models for the speech recognition system. We also report on advances in the technology to integrate a spoken pronunciation with a spoken spelling, in order to improve spelling accuracy. While the original algorithm was designed for a “speak and spell ” input mode, we have shown here that the same methods can be applied to separately uttered spoken and spelled forms of the word. By concatenating the two waveforms, we can take advantage of the mutual constraints realized in an integrated composite FST. Using an OGI corpus of separately spoken and spelled names, we have demonstrated letter error rates of under 6 % for in-vocabulary words and under 11 % for words not contained in the training lexicon, a 44 % reduction in error rate over that achieved without use of the spoken form. We anticipate applying this technique to unknown words embedded in a larger context, followed by solicited spellings. 1.

This paper details our work in developing a technique which can automatically generate class n-gram language models from natural language (NL) grammars in dialogue systems. The procedure eliminates the need for double maintenance of the recognizer language model and NL grammar. The resulting language model adopts the standard class n-gram framework for computational efficiency. Moreover, both the n-gram classes and training sentences are enhanced with semantic/syntactic tags defined in the NL grammar, such that the trained language model preserves the distinctive statistics associated with different word senses. We have applied this approach in several different domains and languages, and have evaluated it on our most mature dialogue systems to assess its competitiveness with preexisting n-gram language models. The speech recognition performances with the new language model are in fact the best we have achieved in both the JUPITER weather domain and the MERCURY flight reservation domain. 1.

This paper explores a new approach to speech recognition in which sub-word units are modeled in terms of linguistic features. Specifically, we have adopted a scheme of modeling separately the manner and place of articulation for these units. A novelty of our work is the use of a generalized definition of place of articulation that enables us to map both vowels and consonants into a common linguistic space. Modeling manner and place separately also allows us to explore a multi-stage recognition architecture, in which the search space is successively reduced as more detailed models are brought in. In the 8,000 word PhoneBook isolated word telephone speech recognition task, we show that such an approach can achieve a recognition WER that is 10% better than that achieved in the best results reported in the literature. This performance gain comes with improvements in search space and computation time as well.

This paper describes research whose ultimate aim is to support automatic entry of new words into a spoken dialogue system through interaction with a user. This research demonstrates an important step towards this goal, through a procedure which integrates information made available via the telephone keypad with a spoken instance of the target word, to produce a candidate spelling and pronunciation for the word. Through the use of a parsing mechanism applied to a 73,000 word proper name lexicon [4], we have been able to create a finite-state transducer (FST) that maps phonetics to graphemics, which can be composed with an FST derived from the keypad input to greatly reduce the search space. Experiments conducted on both the OGI name corpus [2] and a set of enrollment data obtained from our Mercury system [5] validate the procedure.

This paper proposes a layered Finite State Transducer (FST) framework integrating hierarchical supra-lexical linguistic knowledge into speech recognition based on shallow parsing. The shallow parsing grammar is derived directly from the full fledged grammar for natural language understanding, and augmented with top-level n-gram probabilities and phrase-level context-dependent probabilities, which is beyond the standard context-free grammar (CFG) formalism. Such a shallow parsing approach can help balance sufficient grammar coverage and tight structure constraints. The context-dependent probabilistic shallow parsing model is represented by layered FSTs, which can be integrated with speech recognition seamlessly to impose early phrase-level structural constraints consistent with natural language understanding. It is shown that in the JUPITER [1] weather information domain, the shallow parsing model achieves lower recognition word error rates, compared to a regular class n-gram model with the same order. However, we find that, with a higher order top-level n-gram model, pre-composition and optimization of the FSTs are highly restricted by the computational resources available. Given the potential of such models, it may be worth pursing an incremental approximation strategy [2], which includes part of the linguistic model FST in early optimization, while introducing the complete model through dynamic composition.

This paper describes the evaluation methodology and results of the DARPA Communicator spoken dialog system evaluation experiments in 2000 and 2001. Nine spoken dialog systems in the travel planning domain participated in the experiments resulting in a total corpus of 1904 dialogs. We describe and compare the experimental design of the 2000 and 2001 DARPA evaluations. We describe how we established a performance baseline in 2001 for complex tasks. We present our overall approach to data collection, the metrics collected, and the application of PARADISE to these data sets. We compare the results we achieved in 2000 for a number of core metrics with those for 2001. These results demonstrate large performance improvements from 2000 to 2001 and show that the Communicator program goal of conversational interaction for complex tasks has been achieved.

This paper describes the evaluation methodology and results of the 2001 DARPA Communicator evaluation. The experiment spanned 6 months of 2001 and involved eight DARPA Communicator systems in the travel planning domain. It resulted in a corpus of 1242 dialogs which include many more dialogues for complex tasks than the 2000 evaluation. We describe the experimental design, the approach to data collection, and the results. We compare the results by the type of travel plan and by system. The results demonstrate some large differences across sites and show that the complex trips are clearly more difficult.

This paper describes a unified architecture for integrating sub-lexical models with speech recognition, and a layered framework for context-dependent probabilistic hierarchical sublexical modelling. Previous work [1, 2, 3] has demonstrated the effectiveness of sub-lexical modelling using a core context-free grammar (CFG) augmented with context-dependent probabilistic models. Our major motivation for designing a unified architecture is to provide a framework such that probabilistic sublexical components can be integrated with other speech recognition components without sacrificing the flexibilities of their independent developments and configurations. At the same time, we are able to obtain a tightly coupled interface between recognizers and sub-lexical linguistic components. We also present a view of using layered probabilistic models to augment CFGs. It captures context-dependent probabilistic information beyond the standard CFG formalism, and provides the flexibility of developing suitable probabilistic models independently for each sub-lexical layer. Experimental results show that the context-dependent probabilistic hierarchical sub-lexical modelling approach can achieve comparable performance to pronunciation network approaches on utterances that contain only in-vocabulary words, while being able to substantially reduce errors on utterances with previously unseen words. 1.

This paper describes MOKUSEI, an end-to-end Japanese version of our JUPITER weather information system. MOKUSEI delivers weather information over the phone through natural conversation with the user. For the most part, MOKUSEI uses the same components for recognition, understanding, and generation that JUPITER uses, and the database and the semantic frames for the weather information content are also shared. However, MOKUSEI motivated us to redesign our GENESIS generation system, in order to improve the quality of translations of weather reports into Japanese. We also had to develop new ways to transcribe user utterances through morphological analysis. MOKU- SEI is fully functional and has already been used for data collection with about 700 naive users. These data have been used for improvement and evaluation of MOKUSEI. This paper also presents the result of evaluating the current version of MOKUSEI.

This paper describes a phase vocoder based technique for voice transformation. This method provides a flexible way to manipulate various aspects of the input signal, e.g., fundamental frequency of voicing, duration, energy, and formant positions, without explicit extraction. The modifications to the signal can be specific to any feature dimensions, and can vary dynamically over time. There are many potential applications for this technique. In concatenative speech synthesis, the method can be applied to transform the speech corpus to different voice characteristics, or to smooth any pitch or formant discontinuities between concatenation boundaries. The method can also be used as a tool for language learning. We can modify the prosody of the student 's own speech to match that from a native speaker, and use the result as guidance for improvements. The technique can also be used to convert other biological signals, such as killer whale vocalizations, to a signal that is more appropriate for human auditory perception. Our initial experiments show encouraging results for all of these applications. 1.

This paper describes results of an experiment with 9 different DARPA Communicator Systems who participated in the June 2000 data collection. All systems supported travel planning and utilized some form of mixed-initiative interaction. However they varied in several critical dimensions: (1) They targeted different back-end databases for travel information; (2) The used different modules for ASR,NLU,TTS and dialog management. We describe the experimental design, the approach to data collection, the metrics collected, and results comparing the systems. 1.

This paper examines an approach of using lexical stress models to improve the speech recognition performance on spontaneous telephone speech. We analyzed the correlation of various pitch, energy, and duration measurements with lexical stress on a large corpus of spontaneous utterances, and identified the most informative features of stress using classification experiments. We incorporated the stress models into the recognizer first-pass Viterbi search and obtained modest but statistically significant improvements over a state-of-the-art real-time performance on the JUPITER domain. 1.

In this paper, we introduce a pitch detection algorithm that is particularly robust for telephone speech and prosodic modeling. The algorithm uses a logarithmically sampled spectral representation of speech, similar to that in the subharmonic summation approach. Constraints for logF/sub 0/ and /spl Delta/logF/sub 0/ are combined in a dynamic programming search to find an optimum pitch track. The search algorithm is able to find a continuous pitch contour regardless of the voicing status, while a separate voicing decision module computes the probability of voicing per frame. We evaluated the algorithm using the Keele pitch extraction reference database under both studio and telephone conditions. Our algorithm is very robust to channel degradation, and compares favorably to XWAVES under telephone conditions. It also significantly outperforms XWAVES when used for tone classification on a telephone quality Mandarin digit corpus.

Language generation is a fundamental component of dialogue systems. Over the past year, we have developed a new generation module for conversational systems developed at MIT using the GALAXY architecture. Our generator, which we call GENESIS- II, resolves many of the shortcomings of its predecessor, GENE- SIS.GENESIS-II makes it substantially easier for users to specify generation, and the generation output is often of a higher quality. In particular, GENESIS-II has improved the ease and quality of generation in foreign languages (Japanese, Chinese, Spanish) and non-traditional languages (SQL, HTML, speech waveforms) . In this paper, we focus on the more advanced features of our system. 1. Introduction Language generation can be defined as the process of transforming a meaning representation into a target string. In conversational systems, this most obviously means preparing a wellformed string to be spoken to the user, as a response to a question. However, a versatile generation mod...

In this paper we report our efforts in data collection and performance evaluation in support of spoken dialogue system development. We describe two understanding metrics called query density and concept efficiency which can be interpreted on a perutterance basis, but which are measured over the course of a dialogue. We also describe the evaluation infrastructure we have developed to support off-line data processing using our GALAXY client-server architecture [8]. We show how we have used these metrics and mechanisms as part of the development of a spoken dialogue system for air-travel information.

In this paper we present a method for integrating confidence scores into the understanding and dialogue components of a speech understanding system. The understanding component of our system receives an #-best list of recognition hypotheses augmented with word-level confidence scores. The confidence scores are used by the understanding component to hypothesize when words in a recognizer's #-best list have been misrecognized. The understanding component has the ability to predict the semantic class of misrecognized words based on the surrounding context and also to suggest when key words which may have been misunderstood should be re-confirmed by the user. The output of the understanding component is passed onto a dialogue control component which can act on various suggestions made by the understanding component. To evaluate the system, experiments were conducted using the JUPITER weather information system. Evaluation was performed at the understanding level using key-value pair conc...

Abstract. This paper introduces orion, a conversational sys-tem that performs off-line tasks and initiates later contact with a user at a pre-negotiated time. Orion has two major episodes of activity: the enrollment of new tasks and the execution of pending tasks. The task manager periodically checks the pend-ing tasks and updates their status, sending off requests to other servers for information and possibly launching a phone call when a particular task has reached its trigger time. A sepa-rate user interface engages in a dialogue with a user to enroll new tasks and/or update existing tasks. Orion is still in an early stage of its development cycle, but it has introduced sev-eral interesting new research issues, such as continuous state maintenance and contact verification.

This paper describes the generation component of our MERCURY flight reservation conversational system. Generation makes use of the GENESIS-II generation server, which represents a significant redesign from its predecessor, GENESIS. While the main focus is on response generation, we also discuss a variety of other generation needs that are fulfilled by GENESIS-II. These include a paraphrase of the user query back into English and into a flattened electronic form, the paraphrase of the electronic form into a database query language, and the conversion of flight tables into HTML format. 1. INTRODUCTION Over the past two years, we have been developing a conversational system that provides access to real flight information, world wide. The system, called Mercury, which has been implemented within the GALAXY communicator architecture [4], adopts a mixed-initiative dialogue model in which the system prompts for information but the user is not required to comply. In [5], we described the Me...

In this paper we present novel solutions to problems related to barge-in in telephony-based conversational systems. In particular we address recovery from falsely detected barge-in events and a method for signaling to the user that barge-in is disallowed at a particular dialogue state. The mechanisms and signals used to manage turn taking are similar to those in human-human conversation, which makes them easy to understand for users without explanation or prior training. 1. INTRODUCTION In telephony-based spoken language systems, it is desirable to let users interrupt system output at any time, in particular if the output is based on erroneous understanding or contain superfluous information. Thus, enabling barge-in, i.e., the ability for the user to start speaking before system output has ended, can significantly enhance the user experience. However, users' new freedom also poses new challenges. One challenge is sorting out true user barge-in from background noise and nonspeech soun...

MUXING is a telephone-based conversational system that allows users to access weather information in Mandarin Chinese over the telephone. Although MUXING utilizes the same architecture as well as most of the same human language technology components as its English predecessor, JUPITER, some modifications to the system were necessary to account for differences between English and Mandarin Chinese. In addition, the weather database needed to be modified to reflect regions of greater interest to potential Chinese users. This paper describes our system development effort, paying particular attention to Mandarinspecific changes to the original JUPITER system. 1. INTRODUCTION For the past decade, our group has been conducting research leading to the development of conversational systems that enable users to access and manage information using spoken dialogue. In this context, multilinguality has always been an important research topic. Our approach to developing multilingual conversational...

We have previously demonstrated that tone modeling improved speech recognition on a digit corpus [7]. In this work, we further improve tone recognition by normalizing for both tone coarticulation and intonation effects. The tone classification errors on continuous digit strings were reduced by 26.1% from the baseline, when the effects of # # downdrift, phrase boundary and tone coarticulation were normalized. We also applied the same approach to conversational speech from the YINHE domain [6], and obtained similar improvements. The word error rate on spontaneous YINHE data was reduced by 16.5% when a simple fourtone model was applied to resort recognizer 10-best outputs. 1. INTRODUCTION Tone is a natural target for prosodic modeling in tonal languages, because of its important role in lexical access. There are four lexical tones in Mandarin Chinese, each corresponding to a canonical # # contour pattern: "high-level", "high-rising", "low-dipping" and "high-falling". However, tones in...

In early 1997, our group initiated a project to develop JUPITER, a conversational interface that allows users to obtain worldwide weather forecast information over the telephone using spoken dialogue. It has served as the primary research platform for our group on many issues related to human language technology, including telephone-based speech recognition, robust language understanding, language generation, dialogue modeling, and multilingual interfaces. Over a two year period since coming online in May 1997, JUPITER has received, via a toll-free number in North America, over 30000 calls (totaling over 180000 utterances), mostly from naive users. The purpose of this paper is to describe our development effort in terms of the underlying human language technologies as well as other system-related issues such as utterance rejection and content harvesting. We also present some evaluation results on the system and its components.

This paper describes our efforts in designing a twostage recognizer with the objective of developing a multidomain speech understanding system. We envisage one first-stage recognition engine that is domain-independent, and multiple second-stage systems specializing in individual domains. A major novelty in our initial two-stage design is a front-end that incorporates angie-based hierarchical sublexical probability models encapsulated within a finite-state transducer (FST) paradigm. This first stage is a context-dependent syllable-level recognizer which outputs acoustic-phonetic networks to be processed in a second pass. The second stage incorporates higher order linguistic knowledge, from phonological to syntactic and semantic, in a tightly coupled search. This system has yielded up to a 28.5% reduction in understanding error, compared with a single stage context-dependent recognizer which does not use angie-based probabilities. 1. INTRODUCTION Today, telephone-based conversational sy...

This paper explores some issues in designing conversational systems with integrated higher level constraints. We experiment with a configuration that combines a context-dependent acoustic front-end, using MIT's SUMMIT recognizer, with ANGIE, a hierarchical framework that models word substructure and phonological processes, and with TINA, a trainable probabilistic natural language (NL) model. Working in the Jupiter weather domain, we develop a computationally tractable system which incorporates higher level linguistic, prosodic and phonological constraints together in the second of a two-pass strategy. Experiments are evaluated using a new understanding performance metric, and the new integrated system achieves up to 17.1% relative reduction in understanding error and 15.4% reduction in word error. In addition, we investigate the possibilities of a twopass system which relies on the first stage for pruning based on syllable-level constraint, and applies linguistic and prosodic knowledge...

In [9], we introduced the ANGIE framework for modelling speech where morphological and phonological substructures of words are jointly characterized by a context-free grammar and represented in a multi-layered hierarchical structure. In [6], we demonstrated a competitive word-spotter based on the ANGIE framework and presented several results comparing the performance of various sublexical filler models. In the present work, completed as a part of [5], we extend the ANGIE framework to a competitive full continuous speech recognition system. Furthermore, given that ANGIE is based on a context-free framework, we have decided to combine ANGIE with TINA ([8]), a contextfree based framework for natural language understanding, into an integrated system. The integrated system led to a 21.7% reduction in word error rate compared to a baseline word bigram recognizer on ATIS. Numerous issues relating to the construction of the combined system were explored. We have also examined the addition of n...

This paper describes two related systems which provide frameworks for encoding linguistic knowledge into formal rules within the context of a trainable probabilistic model. The first system, TINA [33], drives top-down from sentence level structure, terminating in either words or syllables. Its main purpose is to provide a meaning representation for the sentence. The other system, ANGIE [36], operates bottom-up from phonetic or orthographic units, characterizing the substructure of syllables/words. It provides a framework for both phonological rule modelling and letter-to-sound/sound-to-letter transformations. The two systems logically converge on the syllable or word layer. We have recently been successful in integrating their combined constraint into a recognizer search, achieving considerable improvement in understanding accuracy [9, 23]. In this paper, I will look both toward the past and the future, identifying and motivating the decisions that were made in the design of TINA and ANGIE and the associated rule formalisms, and contemplating various remaining open research issues.

GALAXY is a client-server architecture for accessing on-line in-formation using spoken dialogue that we introduced at ICSLP-94. It has served as the testbed for developing human language technologies for our group for several years. Recently, we have initiated a significant redesign of the GALAXY architecture to make it easier for many researchers to develop their own appli-cations, using either exclusively their own servers or intermixing them with servers developed by others. This redesign was done in part due to the fact that GALAXY has been designated as the first reference architecture for the new DARPA Communicator Program. The purpose of this paper is to document the changes to GALAXY that led to this first reference architecture, which makes use of a scripting language for flow control to provide flexible interaction among the servers, and a set of libraries to support rapid prototyping of new servers. We describe the new reference architecture in some detail, and report on the current status of its development. 1.

Prosodic cues (namely, fundamental frequency, energy and duration) provide important information for speech. For a tonal language such as Chinese, fundamental frequency ( ) plays a critical role in characterizing tone as well, which is an essential phonemic feature. In this paper, we describe our work on duration and tone modeling for telephone-quality continuous Mandarin digits, and the application of these models to improve recognition. The duration modeling includes a speaking-rate normalization scheme. A novel extraction algorithm is developed, and parameters based on orthonormal decomposition of contour are extracted for tone recognition. Context dependency is expressed by "tri-tone" models clustered into broader classes. A 20.0% error rate is achieved for four-tone classification. Over a baseline recognition performance of 5.1% word error rate, we achieve 31.4% error reduction with duration models, 23.5% error reduction with tone models, and 39.2% error reduction with duration and tone models combined.

Telegraphic messages with numerous instances of omission pose a new challenge to parsing in that a sentence with omission causes a higher degree of ambiguity than a sentence without omission. Misparsing induced by omissions has a far-reaching consequence in machine translation. Namely, a misparse of the input often leads to a translation into the target language which has incoherent meaning in the given context. This is more frequently the case if the structures of the source and target languages are quite different, as in English and Korean. Thus, the question of how we parse telegraphic messages accurately and efficiently becomes a critical issue in machine translation. In this paper we describe a technical solution for the issue, and present the performance evaluation of a machine translation system on telegraphic messages before and after adopting the proposed solution. The solution lies in a grammar design in which lexicalized grammar rules defined in terms of semantic categories and syntactic rules defined in terms of part-of-speech are utilized together. The proposed grammar achieves a higher parsing coverage without increasing the amount of ambiguity/misparsing when compared with a purely lexicalized semantic grammar, and achieves a lower degree of ambiguity/misparses without decreasing the parsing coverage when compared with a purely syntactic grammar.

We describe a novel hierarchical duration model for speech recognition. The modelling scheme is based on the angie framework, a flexible unified sublexical representation for speech applications. Our duration model captures contextual factors that influence duration of sublexical units at multiple linguistic levels simultaneously, using both relative and absolute duration information. The modelling procedure involves a normalization scheme which produces a new measure for relative speaking rate at a word level. This may be used to explore phenomena in speech timing and we present studies on secondary effects of speaking rate here. This duration model demonstrates its ability to aid speech recognition in phonetic recognition experiments where it has yielded a relative improvement of up to 7.7%. In word spotting, a study employing duration as a post-processor in disambiguating between 2 acoustically similar keywords reduces relative error by 68%. Furthermore, a fully integrated duration...

We describe our recent work in implementing a word-spotting system based on the ANGIE framework and the effects of varying the nature of the sublexical constraints placed upon the wordspotter 's filler model. ANGIE is a framework for modelling speech where the morphological and phonological substructures of words are jointly characterized by a context-free grammar and are represented in a multi-layered hierarchical structure. In this representation, the upper layers capture syllabification, morphology, and stress, the preterminal layer represents phonemics, and the bottom terminal categories are the phones. ANGIE provides a flexible framework where we can explore the effects of sublexical constraints within a word-spotting environment. Our experiments with spotting city names in ATIS validate the intuition that increasing the constraints present in the model improves performance, from 85.3 FOM for phone bigram to 89.3 FOM for a word lexicon. They also empirically strengthens our belief...

The galaxy system is a human-computer conversational system providing a spoken language interface for accessing on-line information. It was initially implemented for English in travel-related domains, including air travel, local city navigation, and weather. We began an effort to develop multilingual systems within the framework of galaxy several years ago. This paper describes our recent work on porting the system to Mandarin Chinese, including speech recognition, language understanding, and language generation components. Overall, the system produced reasonable responses nearly 70% of the time for spontaneous test data collected in a wizard environment. 1. INTRODUCTION The galaxy system is a client/server architecture for computer conversational systems [1]. In designing galaxy, we drew heavily on experience gained in the development of galaxy's predecessor, voyager [2]. Voyager was not initially designed to easily support multiple languages, but through a trial-and-error process...

This paper describes the discourse component of GALAXY, a multidomain, multimodal conversational system.In designing this module, we are attempting to develop domain-independent mechanisms, controlled via declarative tables, to promote convenientinstantiation of a discourse component for each new domain.Direct anaphoric reference as well as elliptical reference are dealt with appropriately.Users can also refer verbally to items selected via mouse clicks.Cross domain references are particularly challenging, as is the ambiguity problem arising from different case roles for different subdomains.Users often utter fragments, sometimes in response to serverinitiated dialogue exchanges, so an extensive fragment interpretation mechanism is supported.

This paper describes a new system for speech analysis, ANGIE, which characterizes word substructure in terms of a trainable grammar.ANGIE capture morpho-phonemic and phonological phenomena through a hierarchical framework.The terminal categories can be alternately letters or phone units, yielding a reversible letter-tosound/sound-to-letter system.In conjunction with a segment network and acoustic phone models, the system can produce phonemicto-phonetic alignments for speech waveforms.For speech recognition, ANGIE uses a one-pass bottom-up best-first search strategy.Evaluated in the ATIS domain, ANGIE achieved a phone error rate of 36%, as compared with 40% achieved with a baseline phone-bigram based recognizer under similar conditions.ANGIE potentially offers many attractive features, including dynamic vocabulary adaptation, as well as a framework for handling unknown words. £Previous experiments have yielded improved pronunciation accuracy without this layer.

Over the past year, we have begun the development of a new restaurant guide domain, DINEX, which utilizes all of the same core technology as our other GALAXY systems [2].The domain server has been adapted from our CityGuide domain within our GALAXY framework, using the same mechanisms to locate places on the map or to give directions.The system has information on several features for restaurants, such as hours, cuisine, parking availability, and an on-line menu.DINEX knows over 450 restaurants in the Boston area, and can also provide directions from subway stops or well-known landmarks.DINEX makes use of a relational database derived from several on-line sources, including a nationally known travel guide.This paper particularly emphasizes the language tools we have developed to construct the relational database semi-automatically and to derive from user queries the appropriate SQL queries for accessing the database.

In this paper, we describe a reversible letter-to-sound/sound-to-letter generation system based on an approach which combines a rule-based formalism with data-driven techniques. We adopt a probabilistic parsing strategy to provide a hierarchical lexical analysis of a word, including information such as morphology, stress, syllabification, phonemics and graphemics. Long-distance constraints are propagated by enforcing local constraints throughout the hierarchy. Our training and testing corpora are derived from the high-frequency portion of the Brown Corpus (10,000 words), augmented with markers indicating stress and word morphology. We evaluated our performance on both letter-to-sound and sound-to-letter generation in terms of whole word accuracy and phoneme/letter accuracy. Using 26 letters and 52 phonemes with an unseen test set, we achieved a word accuracy of 69.3% and a phoneme accuracy of 91.7% for letter-to-sound generation, and a word accuracy of 51.9% and letter accuracy of 88.6% for sound-to-letter generation.>

A recent extension to the MIT ATIS (Air Travel Information Service) system, which allows it to answer a question when a full linguistic analysis fails is described. Robust parsing is applied only after a full analysis has failed, and it involves the two stages of (1) parsing a set of phrases and clauses, and (2) gluing them together to obtain a single semantic frame encoding the full meaning of the sentence. In a recent evaluation, less than two-thirds of the sentences analyzed yielded a full parse, but the overwhelming majority of the remaining sentences were analyzed correctly by the robust parsing scheme. When text input was replaced by recognizer outputs, even though the recognizer produced greater than 50% sentence error rate, the drop in score (%correct-%incorrect) was only 10 percentage points. This indicates that most of the recognizer errors are harmless in terms of meaning analysis, as long as a robust mechanism for accounting for the parsable phrases is in place.>

The MIT VOYAGER speech understanding system is an urban exploration and navigation system that interacts with the user through spoken dialogue, text, and graphics. The authors describe recent attempts at improving the integration between the speech recognition and natural language components. They used the generation capability of the natural language component to produce a word-pair language model to constrain the recognizer's search space, thus improving the coverage of the overall system. They also implemented a strategy in which the recognizer generates the top N word strings and passes them along to the natural language component for filtering. Results on performance evaluation are presented.>

Early experience with the development of the MIT VOYAGER spoken language system is described, and its current performance is documented. The three components of VOYAGER, the speech recognition component, the natural language component, and the application back-end, are described.>

The phonological modeling and lexical access components of the SUMMIT speech recognition system are described in detail. SUMMIT makes explicit use of acoustic-phonetic knowledge, embedded in a segmental framework that can be trained automatically. Performance results for the complete system on the DARPA 1000-word Naval Resource Management task are presented.>

A natural language system, TINA, which integrates key ideas from context-free grammars, augmented transition networks (ATNs), and unification grammars has been developed. The parser uses a best-first search strategy, with probability assignments on all arcs obtained automatically from a set of example sentences. An initial context-free grammar, derived from the example sentences, is converted to an implicit probabilistic network structure. Control includes both top-down and bottom-up cycles, and key parameters are passed among nodes to deal with long-distance movement and agreement constraints. The probabilities provide a natural mechanism for exploring more common grammatical constructions first. Arc probabilities reduce test-set perplexity sixfold. A strategy for dealing with movement that can handle nested and chained gaps efficiently and rejects crossed gaps is introduced.>

Recently, the authors initiated a project to develop a phonetically-based spoken-language-understanding system called SUMMIT. In contrast to many of the past efforts that make use of heuristic rules whose development requires intense knowledge engineering, their approach attempts to express the speech knowledge within a formal framework using well-defined mathematical tools. In the authors' system, features and decision strategies are discovered and trained automatically, using a large body of speech data. The authors describe those parts of the system dealing with acoustic segmentation and phonetic classification and document its current performance.>

A computer system for speech analysis based on human auditory processing is described. The system consists of a bank of 40 independent channels, spanning the frequency range from 130 to 6400 Hz. Each channel includes a linear critical band filter followed by a model for the transformation from basilar membrane motion to nerve fiber response, incorporating such nonlinear effects as half-wave rectification, adaptation, spontaneous response, and saturation. The output of this stage includes the detailed waveshape of each cycle of the probabilistic response, but the data are never reduced to a spike sequence. Finally, each channel is subjected independently to both envelope detection and synchrony detection, where the latter is tuned to the characteristic frequency [CF] of the associated filter. It is likely that the envelope response will be useful for marking acoustic boundaries and for making broad category decisions, whereas the synchronous response is more appropriate for making fine distinctions.

Over the past five years, a number of useful speech- and language-related research tools have been developed at MIT. These tools are aids for efficiently analyzing the acoustic characteristics of speech and the phonological properties of a language. They are playing a valuable role in our own research, as well as in research conducted elsewhere. This paper describes several of the systems which have been developed for use on our Lisp Machine workstations.

This paper describes a system for processing sonorant regions of speech, motivated by knowledge of the human auditory system. The spectral representation is intended to reflect a proposed model for human auditory processing of speech, which takes advantage of synchrony in the nerve firing patterns to enhance formant peaks. The auditory model is also applied to pitch extraction, and thus a temporal pitch processor is envisioned. The spectrum is derived from the outputs of a set of linear fillers with critical bandwidths. Saturation and adaptation are incorporated for each filter independently. Each "spectral" coefficient is determined by weighting the amplitude response at that frequency by a measure of synchrony to the center frequency of the filter. Pitch is derived front a waveform generated by adding the rectified filter outputs across the frequency dimension. The spectral estimator and the pitch estimator are illustrated by processing pure tones and natural speech.

Integrated packet-switched networks have potential for providing improved performance by dynamically sharing transmission bandwidths between various users and user types, but new flow control methods are needed to deal with packetized voice traffic. This paper describes a packet voice flow control concept based on embedded speech coding. Results are presented from a computer simulation study of the technique in the context of a multilink wideband packet speech network. Several control methodologies are described, leading to an end-to end feedback approach that achieves stable operation and efficient utilization of network resources by adaptively matching transmitted voice bit rates to prevailing network conditions. Issues in the design of embedded speech coding algorithms are reviewed and a candidate structure based on channel vocoding principles is presented, along with the subjective results of some preliminary listening tests