Spoken language understanding (SLU) in today’s conversational systems focuses on recognizing a set of domains, intents, and associated arguments, that are determined by application developers. User requests that are not covered by these are usually directed to search engines, and may remain unhandled. We propose a method that aims to find common user intents amongst these uncovered, out-of-domain utterances, with the goal of supporting future phases of dialog system design. Our approach relies on finding common semantic patterns in uncovered user utterances using an Abstract Meaning Representation based semantic parser. We represent the corpus as a graph and find subgraphs that represent clusters, by pruning the corpus graph according to frequency and entropy. We employ crowd-workers to select and label the resulting clusters and compare resulting clusters with two baselines. Experimental analyses show that we obtain higher coverage and accuracy with the semantic parsing based clustering method. Furthermore, since the intents and candidate slots are already induced, these utterances can also be used in unsupervised SLU modeling. In intent classification experiments, we show that the statistical model trained using the clusters formed by this approach results in higher classification F-measure (showing about 25% relative improvement) in comparison to the alternatives.

This paper presents the design and implementation of a programming system that enables third-party developers to add spoken natural language (SNL) interfaces to standalone mobile applications. The central challenge is to create statistical recognition models that are accurate and resource-efficient in the face of the variety of natural language, while requiring little specialized knowledge from developers. We show that given a few examples from the developer, it is possible to elicit comprehensive sets of paraphrases of the examples using internet crowds. The exhaustive nature of these paraphrases allows us to use relatively simple, automatically derived statistical models for speech and language understanding that perform well without per-application tuning. We have realized our design fully as an extension to the Visual Studio IDE. Based on a new benchmark dataset with 3500 spoken instances of 27 commands from 20 subjects and a small developer study, we establish the promise of our approach and the impact of various design choices.

Conversing on cell phones while driving is a risky, yet commonplace activity. State legislatures in the U.S. have enacted rules that limit hand-held phone conversations while driving but that allow for hands-free conversations. However, studies have demonstrated that the cognitive load of conversation is a significant source of distraction that increases the likelihood of accidents. We explore in a controlled study with a driving simulator the effectiveness of proactive alerting and mediation of communications during phone conversations while driving. We study the use of auditory messages indicating upcoming critical road conditions and placing calls on hold. We found that such actions reduce driving errors and that alerts sharing details about situations were more effective than general alerts. Drivers found such a system valuable in most situations for maintaining driving safety. These results provide evidence that context-sensitive mediation systems could play a valuable role in focusing drivers' attention on the road during phone conversations.

The Utterance Classification (UC) method has become a developer’s choice over traditional Context Free Grammars (CFGs) for voice menus in telephony applications. This data driven method achieves higher accuracy and has great potential to utilize a huge amount of labeled training data. But, having a human manually label the training data can be expensive. This paper provides a robust recipe for training a UC system using inexpensive acoustic data with limited transcriptions or semantic labels. It also describes two new algorithms that use caller confirmation, which naturally occurred within a dialog, to generate pseudo semantic labels. Experimental results show that, after having sufficient labeled data to achieve a reasonable accuracy, both of our algorithms can use unlabeled data to achieve the same performance as a system trained with labeled data, while completely eliminating the need for human supervision. Index Terms: Call Routing, Statistical grammars, Spoken language understanding (SLU), Utterance Classification (UC)

Conversing on cell phones while driving an automobile is a common practice. We examine the interference of the cognitive load of conversational dialog with driving tasks, with the goal of identifying better and worse times for conversations during driving. We present results from a controlled study involving 18 users using a driving simulator. The driving complexity and conversation type were manipulated in the study, and performance was measured for factors related to both the primary driving task and secondary conversation task. Results showed significant interactions between the primary and secondary tasks, where certain combinations of complexity and conversations were found especially detrimental to driving. We present the studies and analyses and relate the findings to prior work on multiple resource models of cognition. We discuss how the results can frame thinking about policies and technologies aimed at enhancing driving safety.

Voice search technology has been successfully applied to help drivers reply SMS messages in automobiles, in which a predefined SMS message template set is searched with ASR hypotheses to form the reply candidate list. In order to efficiently organize the SMS message template set and improve the quality of the reply candidate list, we proposed to apply n-gram translation model and logistic regression to detect paraphrase SMS messages. Both of the proposed algorithms outperform the edit distance based paraphrase detection baseline, brining 40.9% and 50.5% EER reduction (relative), respectively.

This paper addresses the problem of using unstructured queries to search a structured database in voice search applications. By incorporating structural information in music metadata, the end-to-end search error has been reduced by 15% on text queries and up to 11% on spoken queries. Based on that, an HMM sequential rescoring model has reduced the error rate by 28% on text queries and up to 23% on spoken queries compared to the baseline system. Furthermore, a phonetic similarity model has been introduced to compensate speech recognition errors, which has improved the end-to-end search accuracy consistently across different levels of speech recognition accuracy.

Automotive infotainment systems now provide drivers the ability to hear incoming Short Message Service (SMS) text messages using text-to-speech. However, the question of how best to allow users to respond to these messages using speech recognition remains unsettled. In this paper, we propose a robust voice search approach to replying to SMS messages based on template matching. The templates are empirically derived from a large SMS corpus and matches are accurately retrieved using a vector space model. In evaluating SMS replies within the acoustically challenging environment of automobiles, the voice search approach consistently outperformed using just the recognition results of a statistical language model or a probabilistic context-free grammar. For SMS replies covered by our templates, the approach achieved as high as 89.7 % task completion when evaluating the top five reply candidates. Index Terms: SMS, information retrieval, voice UI, voice search 1.

Speech recognition technology is prone to mistakes, but this is not the only source of errors that cause speech recognition systems to fail; sometimes the user simply does not utter the command correctly. Usually, user mistakes are not considered when a system is designed and evaluated. This creates a gap between the claimed accuracy of the system and the actual accuracy perceived by the users. We address this issue quantitatively in our in-car infotainment media search task and propose expanding the capability of voice command to accommodate user mistakes while retaining a high percentage of the performance for queries with correct syntax. As a result, failures caused by user mistakes were reduced by an absolute 70 % at the cost of a drop in accuracy of only 0.28%.

Live search for mobile is a cellphone application that allows users to interact with Web-based information portals. Currently the implementation is focused on information related to local businesses: their phone numbers and addresses, directions, reviews, maps of the surrounding area, and traffic. This paper describes a speech-recognition interface which was recently developed for the application, which allows the users to interact by voice. The paper presents the overall architecture, the user interface, the design and implementation of the speech recognition grammars, and initial performance results indicating that for sentence level utterance recognition we achieve 60 to 65% of human capability.

This paper presents a novel approach to language modeling for voice search based on the idea and method of statistical machine translation. We propose an n-gram based translation model that can be used for listing-to-query translation. We then leverage the query forms translated from listings to improve language modeling. The translation model is trained in an unsupervised manner using a set of transcribed voice search queries. Experiments show that the translation approach yielded drastic perplexity reductions compared with a baseline language model where no translation is applied.

In this paper we address two related challenges in multimodal local search applications on mobile devices: first, correctly displaying the business names, and second, harvesting language model training data from an inconsistently labeled corpus. We investigate the impact of common text normalization and the quality of language model training corpus on the accuracy of displayed results. We propose a new language model framework that eliminates the need for explicit inverse text normalization. The same framework can be applied to sift through corrupted language model training data. Our new language model is 25 % more accurate while 25 % smaller in size. Index Terms: text normalization, inverse text normalization, language model, multimodal, voice search, transduction,

This paper presents an Automatic Reading Tutoring (ART) system using state-of-the-art speech recognition technologies aimed to improve children’s oral reading ability. The features of this system include a compact and robust language model designed for detecting disfluencies in children’s speech, low-footprint implementation, and built-in microphone array. Our system is targeting on hand-held devices to provide better accessibility, flexibility, and freedom for children’s reading practice. The focus of this paper is on the current system’s architecture, which has achieved real-time performance on two hand-held, small-form-factor devices (UMPC and Motion Tablet), with the same detection rate and false alarm rate as on desktop PCs. We also report the latest effort on a prototype system running on a PDA (Windows Mobile 6). Index Terms: Automatic Reading Tutoring (ART) system, children ’ speech recognition, hand-held devices 1.

Voice search applications encourage users to “just say what you want ” in order to obtain useful mobile content such as automated directory assistance (ADA). Unfortunately, when users only remember part of what they are looking for, they are forced to guess, even though what they know may be sufficient to retrieve the desired information. In this paper, we propose expanding the capabilities of voice search to allow users to explicitly express their uncertainties as part of their queries, and as such, to provide partial knowledge. Applied to ADA, we highlight the enhanced user experience uncertain expressions afford and delineate how we performed language modeling and information retrieval. We evaluate our approach by assessing its impact on overall ADA performance and by discussing the results of an experiment in which users generated both uncertain expressions as well as guesses for directory listings. Uncertain expressions reduced relative error rate by 31.8 % compared to guessing. Index Terms: voice search, user uncertainty, something 1.

Internet usage on mobile devices continues to grow as users seek anytime, anywhere access to information. Because users frequently search for businesses, directory assistance has been the focus of many voice search applications utilizing speech as the primary input modality. Unfortunately, mobile settings often contain noise which degrades performance. As such, we present Search Vox, a mobile search interface that not only facilitates touch and text refinement whenever speech fails, but also allows users to assist the recognizer via text hints. Search Vox can also take advantage of any partial knowledge users may have about the business listing by letting them express their uncertainty in an intuitive way using verbal wildcards. In simulation experiments conducted on real voice search data, leveraging multimodal refinement resulted in a 28% relative reduction in error rate. Providing text hints along with the spoken utterance resulted in even greater relative reduction, with dramatic gains in recovery for each additional character.

Recently, there has been a rapidly increasing interest in using ASR for children's language learning. An automatic reading tutor system built with ASR technologies can track children's oral reading against story texts, detect reading miscues, and measure the level of reading fluency. They may even diagnose the nature of the miscues and provide feedback to improve reading skills. In such tasks, N-gram language models (LM) may be trained from the whole story text, or may be generated based on current story sentence with heuristic probabilities for both regular words in the sentence and explicitly predicted reading miscues. The disadvantages of those methods are either they require a relatively large text and are time-consuming, or a large-sized LM and complex processing are needed to accommodate all possible words in reading stories as well as in reading miscues. This paper proposes an efficient and robust LM which can be easily built on-the-fly with current reading sentences. With an additional parallel "garbage" model, the LM can also deal effectively with a wide range of reading miscues. Our experiments in a standard children's reading task show that the new LM reaches the state-of-the-art performance in detecting reading miscues with a fast speed while only a relatively simple children's acoustic model of speech was used.

Automated Directory Assistance (ADA) allows users to request telephone or address information of residential and business listings using speech recognition. Because callers often express listings differently than how they are registered in the directory, ADA systems require transcriptions of alternative phrasings for directory listings as training data, which can be costly to acquire. As such, a framework in which data can be contributed voluntarily by large numbers of Internet users has tremendous value. In this paper, we introduce People Watcher, a computer game that elicits transcribed, alternative user phrasings for directory listings while at the same time entertaining players. Data generated from the game not only overlapped actual audio transcriptions, but resulted in a statistically significant 15% relative reduction in semantic error rate when utilized for ADA. Furthermore, semantic accuracy was not statistically different than using the actual audio transcriptions. Index Terms: game, automated directory assistance 1.

The availability of digital maps and mapping software has led to significant growth in location-based software and services. To safely use these applications in mobile and automotive scenarios, users must be able to input precise locations using speech. In this paper, we propose a novel method for location understanding based on spoken intersections. The proposed approach utilizes a rich, automatically-generated grammar for street names that maps all street name variations into a single canonical semantic representation. This representation is then transformed to a sequence of position-dependent subword units. This sequence is used by a classifier based on the vector space model to reliably recognize spoken intersections in the presence of recognition errors and incomplete street names. The efficacy of the proposed approach is demonstrated using data collected from users of a deployed spoken dialog system.

Currently there are no dialog systems that enable purely voice-based access to the unstructured information on websites such as Wikipedia. Such systems could be revolutionary for non-literate users in the developing world. To investigate interface issues in such a system, we developed VoicePedia, a telephone-based dialog system for searching and browsing Wikipedia. In this paper, we present the system, as well as a user study comparing the use of VoicePedia to SmartPedia, a Smartphone GUI-based alternative. Keyword entry through the voice interface was significantly faster, while search result navigation, and page browsing were significantly slower. Although users preferred the GUI-based interface, task success rates between both systems were comparable – a promising result for regions where Smartphones and data plans are not viable. Index Terms: dialog system, information access

Voice search is the technology underlying many spoken dialog applications that enable users to access information using spoken queries. This paper reviews voice search technology, and proposes a new and effective method for computing semantic confidence measures. It explores the use of maximum entropy classifiers as confidence models, and investigates a feature selection algorithm that leads to an effective subset of prominent features for the classifier. The experimental results on a directory assistance application show that the reduced feature set not only makes the model more effective in handling different recognition and search engine combinations, but also results in a very informative confidence measure that is closely correlated with the actual voice search accuracy. Index Terms: voice search, directory assistance, confidence measure, Tf-Idf vector space model, maximum entropy model. 1.

The automated directory assistance system (ADAS) is traditionally formulated as an automatic speech recognition (ASR) problem. Recently, it has been formulated as a voice search problem, where a spoken utterance is firstly converted into text, which in turn is used to search for the listing. In this paper, we focus on the design and development of the utterance-to-listing component of ADAS. We show that many theoretical and practical issues need to be resolved when applying the basic idea of voice search to the development of ADAS. We share our experiences in addressing these issues, especially in pre-processing the listing database, generating a high performance LM, and developing efficient, accurate, and robust search algorithms. Field tests of our prototype system indicate that an 81 % task completion rate can be achieved. Index Terms: speech recognition, directory assistance, voice search, TFIDF, spoken dialog system, vector space model

Voice-Rate is an automated dialog system which provides access to over one million ratings of products and businesses. By calling a toll-free number, consumers can access ratings for products, national businesses such as airlines, and local businesses such as restaurants. Voice-Rate also has a facility for recording and analyzing ratings that are given over the phone. The service has been primed with ratings taken from a variety of web sources, and we are augmenting these with user ratings. Voice-Rate can be accessed by dialing 1-877-456-DATA. 1

We propose a technique for rapid speech application development that generates robust semantic context-free grammars (CFG) given rigid CFGs as input. Users' speech does not always conform to rigid CFGs, so robust grammars improve the caller's experience. Our system takes a simple CFG and then generates a hybrid n-gram/CFG that is written in the W3C SRGS format and thus can run in many standard automatic speech recognition engines. The hybrid network leverages an application-independent word n-gram which can be shared across different applications. In addition, our tool allows developers to provide a few example sentences to adapt the n-gram for improved accuracy. Our experiments show the robust CFG has no loss in accuracy for test utterances that can be covered by the rigid CFG, but offers large improvements for cases where the user's sentence cannot be covered by the rigid CFG. It also has a much better rejection for utterances that contain no slot at all. With a few example sentences for adaptation, our robust CFG can achieve the recognition accuracy close to the class-based n-gram LM customized for the application.

This paper reports our recent development of a highly reliable call analysis technique that makes novel use of automatic speech recognition (ASR), speech utterance classification and non-speech features. The main ideas include the use the N-Gram filler model to improve the ASR accuracy on important words in a message, and the integration of recognized utterance with non-speech features such as utterance length, and the use of utterance classification technique to interpret the message and extract additional information. Experimental evaluation shows that the use of the utterance length, recognized text, and the classifier’s confidence measure reduces the classification error rate to 2.5 % of the test sets.

In this paper we propose a novel, effective, and efficient utterance verification (UV) technology for access control in the interactive voice response (IVR) systems. The key of our approach is to construct a context-free grammar by using the secret answer to a question and a word N-gram based filler model. The N-gram filler provides rich alternatives to the secret answer and can potentially improve the accuracy of the UV task. It can also absorb carrier words used by callers and thus can improve the robustness. We also propose using a predictor based on the best alternative to calculate the confidence. We show detailed experimental results on a tough UV test set that contains 930 positive and 930 negative cases and discuss types of questions that are suitable for the UV task. We demonstrate that our approach can achieve a 2.14% equal error rate (EER) on average and 0.8 % false accept rate if the false reject rate is 2.6 % and above. This is a 49 % EER reduction compared with the approaches using acoustic fillers, and a 72 % EER reduction compared with the posterior probability based confidence measurement. Index Terms: utterance verification, filler model, word spotting, confidence measure

In this paper we describe our solutions to the problems faced in building the fully featured auto-attendant application that currently provides telephone access to approximately 50,000 Microsoft employees. The paper focuses on issues that the application developer can address directly. Some of the problems tackled in building the grammar include the handling of homonyms, several people with the same name, and data requiring name normalization. Our voice UI helped with this high perplexity task by providing multiple ways to specify the required employee, including spelling their email address or selecting them from a list of employees. Over 70% of the calls analyzed were successfully completed. The system was built using the Microsoft Speech Application SDK and runs on the Microsoft Speech Server 2004.

To bring speech recognition mainstream, researchers have been working on automatic grammar development tools. Most of the work focused on the modeling of sentence level commands for mixed-initiative dialogs. In this paper we describe a novel approach that enables the developers with little grammar authoring experience to construct high performance speech grammars for alphanumeric concepts, which are often needed in the more commonly used directed dialog systems in practice. A developer can simply write down a regular expression for the concept and the algorithm automatically constructs a W3C grammar with appropriate semantic interpretation tags. While the quality of the grammar is ultimately determined by the way in which the regular expression is written, the algorithm relieves the developers from the difficult tasks of optimizing grammar structures and assigning appropriate semantic interpretation tags, thus it greatly speeds up grammar development and reduces the requirement of expertise. Preliminary experimental results have shown that the grammar created with this approach consistently outperformed the general alphanumeric rules in the grammar library. In some cases the semantic error rates were cut by more than 50%. 1.

Input of Asian ideographic characters has traditionally been one of the biggest impediments for information processing in Asia. Speech is arguably the most effective and efficient input method for Asian non-spelling characters. This paper presents a Japanese large-vocabulary continuous speech recognition system based on Microsoft Whisper technology. We focus on the aspects of the system that are language specific and demonstrate the adaptability of the Whisper system to new languages. In this paper, we demonstrate that our pronunciation/part-of-speech distinguished morpheme based language models and Whisper based Japanese senonic acoustic models are able to yield state-of-the-art Japanese LVCSR recognition performance. The speaker-independent character and Kana error rates on the JNAS database are 10% and 5% respectively.

The Whistler text-to-speech engine was designed so that we can automatically construct the model parameters from training data. This paper focuses on the improvements on prosody and acoustic modeling, which are all derived through the use of probabilistic learning methods. Whistler can produce synthetic speech that sounds very natural and resembles the acoustic and prosodic characteristics of the original speaker. The underlying technologies used in Whistler can significantly facilitate the process of creating generic TTS systems for a new language, a new voice, or a new speech style. Whisper TTS engine supports Microsoft Speech API and requires less than 3 MB of working memory.

This paper describes new algorithms for finding the K-most critical paths, checking static sensitizability of these paths, and performing incremental timing verification on combinational circuits. The static sensitization method uses binary decision diagrams to avoid costly backtracking operation used in other path analysis programs. The speed and efficiency of the techniques are demonstrated individually using the ISCAS benchmark circuits, and then together in a timing optimization loop.

An efficient algorithm for the indentification of viable paths in a combinational logic circuit using binary decision diagrams is described. The viable paths are justified by generating and resolving logic and delay constraints along the critical path based on the stable times and stable values of the side inputs. Results of the analysis, using the ISCAS combinational benchmark circuits, indicate that most of the circuits can be analyzed in a few CPU-seconds.>

A systematic methodology for automatic consistency checking and optimization of parameterized macromodels is described. An overview of the entire macromodel verification system. iMAVERICK, is provided. An efficient stochastic algorithm to optimize the parameters of a given macromodel is described, and a number of characteristics of the algorithm are examined. A heuristic measure of the likelihood that a cost evaluation will provide useful information or a better solution that is used in the algorithm is based on the information accumulated from the previous cost evaluations. This heuristic measure is used to reject poor sample points without sacrificing the quality of solutions. A waveform comparison technique for evaluating the performance of the proposed configuration in the optimization process is described. It is sensitive to phase errors between waveforms and has the capability of filtering out glitches and high-frequency noise. Several examples of macromodel optimization using the iMAVERICK system are presented.>

A description is presented of efforts applying mixed-mode simulation techniques to two other areas of research. In incremental simulation, techniques are presented to perform fast incremental circuit simulation based on a modified incremental-in-space approach and event-driven techniques. In fault simulation, a mixed-mode fault simulator is presented that allows the user to specify any type of electrical level fault at the transistor level, as opposed to one of the simple stuck-at faults used in logic simulators. The program performs fault simulation, using mixed-mode techniques, and provides the fault coverage of a set of input patterns. To improve efficiency, concurrent fault simulation with a table look-up scheme is used. The merits of both algorithms are demonstrated with simulation results that show significant speed-ups over standard approaches.>