Topic distribution matrices created by topic models are typically used for document classification or as features in a separate machine learning algorithm.Existing methods for evaluating these topic distributions include metrics such as coherence and perplexity; however, there is a lack of statistically grounded evaluation tools.We present a statistical method for investigating group difference in the documenttopic distribution vectors created by latent Dirichlet allocation (LDA).After transforming the vectors using Aitchison geometry, we use multivariate analysis of variance (MANOVA) to compare sample means and calculate effect size using partial eta-squared.We report the results of validating this method on a subset of the 20Newsgroup corpus.We also apply this method to a corpus of dialogues between Autistic and Typically Developing (TD) children and trained examiners.We found that the topic distributions of Autistic children differed from those of TD children when responding to questions about social difficulties.Furthermore, the examiners' topic distributions differed between the Autistic and TD groups when discussing emotions and social difficulties.These results support the use of topic modeling in studying clinically relevant features of social communication such as topic maintenance.

We explore the feasibility of using pupil diameter to estimate how the cognitive load of the driver changes during a spoken dialogue task with a remote conversant. The conversants play a series of Taboo games, which do not follow a structured turn-taking nor initiative protocol. We contrast the driver's pupil diameter when the remote conversant begins speaking with the diameter right before the driver responds. Although we find a significant difference in pupil diameter for the first pair in each game, subsequent pairs show little difference. We speculate that this is due to the less structured nature of the task, where there are no set time boundaries on when the conversants work on the task. This suggests that spoken dialogue systems for in-car use might better manage the driver's cognitive load by using a more structured interaction, such as system-initiative dialogues.

In a driving simulator study, we explore the feasibility of using pupil diameter to estimate how the cognitive load of the driver changes during a spoken dialogue with a remote conversant. We confirm that it is feasible to use pupil diameter to differentiate between parts of the dialogue that increase the cognitive load of the driver, and those that decrease it. Our long term goal is to build a spoken dialogue system that can adapt its behavior when the driver is under high cognitive load, whether from the driving task or the dialogue task. Index Terms: dialog, cognitive load, pupil diameter, driving 1.

Reinforcement learning is becoming a popular tool for building dialogue managers. This paper addresses two issues in using RL. First, we propose two methods for finding MDP violations. Both methods make use of computing Q scores when testing the policy. Second, we investigate how convergence happens. To do this, we use a dialogue task in which the only source of variability is the dialogue policy itself. This allows us to study how and when convergence occurs as training progresses. The work in this paper will help dialogue designers build effective policies and understand how much training is necessary. 1.

Speakers strive to minimize inter-turn gaps when engaged in a dialogue. However, little work has addressed what impact this might have on the fluency of the following speech. In this paper we explore whether there are interactions between turn-taking gaps and turn-initial disfluencies and if the social pressure to respond to questions plays a role in that interaction. Our results indicate that child speakers are more likely to become disfluent both after a question and as the gap length increases, and that the two interact to further increase the likelihood. We also compared the speech of children with Typical Development (TD) to those with Autism Spectrum Disorder (ASD) or Developmental Language Disorder (DLD), where we found that those with ASD were less likely to become disfluent after a question. This finding suggests that the trade-off between timing and disfluencies is driven by social obligation, and that speakers are willing to tolerate disfluencies so as to maintain a short delay. Index Terms: turn-taking gaps, disfluencies, social pressure, language impairments

We present computer tools to help speech-language pathologists (SLP) in counting disfluencies, for both real-time and transcriptbased counts. The latter tend to be more precise and show which words are involved in each disfluency. Our approach allows real-time counts to be used as the basis for transcript-based counts. We employ automatic speech recognition to generate a word transcript (for read-speech samples), and then automatically merge the disfluency annotations with the word transcript, and have the SLP review parts of the audio file where a disfluency annotation was placed. This approach results in improved disfluency annotations, and can be done in 3 times real-time. Index Terms: stuttering, disfluency counts, user-interface 1.

In this article we focus on human–human multi-tasking dialogues, in which pairs of conversants, using speech, work on an ongoing task while occasionally completing real-time tasks. The ongoing task is a poker game in which conversants need to assemble a poker hand, and the real-time task is a picture game in which conversants need to find out whether they have a certain picture on their displays. We employ empirical corpus studies and machine learning experiments to understand the mechanisms that people use in managing these complex interactions. First, we examine task interruptions: switching from the ongoing task to a real-time task. We find that generally conversants tend to interrupt at a less disruptive context in the ongoing task when possible. We also find that the discourse markers oh and wait occur in initiating a task interruption twice as often as in the conversation of the ongoing task. Pitch is also found to be statistically correlated with task interruptions; in fact, the more disruptive the task interruption, the higher the pitch. Second, we examine task resumptions: returning to the ongoing task after completing an interrupting real-time task. We find that conversants might simply resume the conversation where they left off, but sometimes they repeat the last utterance or summarize the critical information that was exchanged before the interruption. Third, we apply machine learning to determine how well task interruptions can be recognized automatically and to investigate the usefulness of the cues that we find in the corpus studies. We find that discourse context, pitch, and the discourse markers oh and wait are important features to reliably recognize task interruptions; and with non-lexical features one can improve the performance of recognizing task interruptions with more than a 50% relative error reduction over a baseline. Finally, we discuss the implication of our findings for building a speech interface that supports multi-tasking dialogue.

We report on the design of spoken tasks for a study that explored how people manage spoken multi-threaded dialogues while one of the conversants is operating a simulated vehicle. Based on a series of preliminary studies we propose a set of considerations that researchers should take into account when designing such tasks. Using these considerations, we discuss two spoken tasks, the parallel twenty questions game and the last letter game, and discuss the successful utilization of these tasks in a study exploring human-human dialogue behavior.

We report on the results of a study in which pairs of subjects were involved in spoken dialogues and one of the subjects also operat-ed a simulated vehicle. We estimated the driver’s cognitive load based on pupil size measurements from a remote eye tracker. We compared the cognitive load estimates based on the physiological pupillometric data and driving performance data. The physiologi-cal and performance measures show high correspondence suggest-ing that remote eye tracking might provide reliable driver cogni-tive load estimation, especially in simulators. We also introduced a new pupillometric cognitive load measure that shows promise in tracking cognitive load changes on time scales of several seconds.

Most applications of reinforcement learning (RL) for dialogue have focused on slot-filling tasks. In this paper, we explore a task that requires negotiation, in which conversants need to exchange information in order to decide on a good solution. We investigate what information should be included in the system's RL state so that an optimal policy can be learned and so that the state space stays reasonable in size. We propose keeping track of the decisions that the system has made, and using them to constrain the system's future behavior in the dialogue. In this way, we can compositionally represent the strategy that the system is employing. We show that this approach is able to learn a good policy for the task. This work is a first step to a more general exploration of applying RL to negotiation dialogues.

In this paper we conduct an exploratory experiment on context restoration in multi-tasking dialogue and report our preliminary findings. We examine a corpus of human-human dialogues, in which pairs of conversants, using speech, work on an ongoing task while occasionally completing real-time tasks. We investigate whether the conversants, when returning to the ongoing task, make any effort to restore the context. First, we identify two types of actions, utterance restatement and information review, as possible restorations. Second, from a statistical analysis, we find that these actions are used more often when returning to the ongoing task, and hence seem to play a role in context restoration. Our findings will help to build a foundation for future speech interfaces that support multi-tasking dialogue.

Abstract This paper describes DialogueView, a tool for annotating dialogues with utterance boundaries, speech repairs, speech act tags, and hierarchical discourse blocks. The tool provides three views of a dialogue: WordView, which shows the transcribed words time-aligned with the audio signal; UtteranceView, which shows the dialogue line-by-line as if it were a script for a movie; and BlockView, which shows an outline of the dialogue. The different views provide different abstractions of what is occurring in the dialogue. Abstraction helps users focus on what is important for different annotation tasks. For example, for annotating speech repairs, utterance boundaries, and overlapping and abandoned utterances, the tool provides the exact timing information. For coding speech act tags and hierarchical discourse structure, a broader context is created by hiding such low-level details, which can still be accessed if needed. We find that the different abstractions allow users to annotate dialogues more quickly without sacrificing accuracy. The tool can be configured to meet the requirements of a variety of annotation schemes.

(1) a reparandum; �ßÞ � �ßÞ��ßÞ� (2) an interruption point, which is where the reparandum ends; (3) optional editing terms, such as ‘um ’ and In previous work, we presented an annotation scheme that can ‘let’s see’; and (4) an alteration, which is the replacement for describe complex disfluencies. In this paper, we first show the the reparandum. This scheme identifies all of the words in-prevalence of complex disfluencies and illustrate the types of volved in a disfluency and the role that each plays. However, it distinctions that our scheme allows. Second, we present an an-does not address multi-iteration repetitions nor clustered disflunotation tool that allows the scheme to be easily applied. Third, encies, which are common in stuttered speech [5, 6]. we present the results of a reliability study in annotating complex disfluencies with the annotation tool. We find that subjects, we can go t- um let’s see can walk to the store even with a minimal amount of training, achieve high intercoder reparandum ip editing term alteration agreement. This work will help pave the way for speech recog-Figure 1: Four parts of repetitions and revisions nizers to precisely model the structure of disfluencies, both for understanding conversational speech of non-stutterers and for Shriberg [7] extended the above scheme so it could describe assessing stuttering severity. complex disfluency patterns. She allows the reparandum and al-Index Terms: disfluencies, stuttering, annotation scheme teration of a disfluency to be embedded inside of another disfluency, in which the outer disfluency uses the alteration (but not 1.

We envision that next-generation spoken dialogue systems will be mixed-initiative. However, it is unclear how exactly a mixed-initiative strategy should be designed; under what circumstances should the system take the initiative, and under what circumstances should it let the user do so. The initiative strategies used in human-human conversation are a good starting point, because they are natural for the user to follow. Studying human-human conversation, however, only gives a descriptive account of human strategies. In this paper, we explore the use of computer simulation to better understand human conventions and give an explanatory account. We have two software agents solve a collaborative task using different initiative strategies, the first derived from analysis of human-human dialogues, and two alternatives based on proposals in the literature. Our simulation results show that the former is more efficient than the others. This helps support the explanation that people use an initiative strategy that minimizes collaborative effort.

In this paper, we present an annotation scheme for disfluencies. Unlike previous schemes, this scheme allows complex disfluencies with multiple backtracking points to be annotated, which are common in stuttered speech. The scheme specifies each disfluency in terms of word-level annotations, thus making the scheme useful for building sophisticated language models of disfluencies. As determining the annotation codes is quite difficult, we have developed a pen and paper procedure in which the annotator lines up the words into rows and columns, from which it is straight-forward for the annotator to determine the annotation tags. Index Terms:: disfluencies, stuttered speech, annotation scheme 1.

To accomplish a task that requires collaboration, people would first agree on a strategy and then together carry it out [1]. Our research interest lies in understanding how people explore different strategies and reach an agreement in conversation. We began by examining two-person dialogues in a very limited domain, in which we could just focus on the agreement process. In this paper, we describe an annotation scheme of coding the conversants ’ behaviors of exploring possible strategies, suggesting and accepting the optimal one, and then maintaining it. We report the inter-coder reliability of the annotation scheme on three expert annotators and two non-experts. Index Terms: annotation scheme, agreement, inter-coder reliability, dialogue.

In this paper, we explore the conventions that people use in managing multiple dialogue threads. In particular, we focus on where in a thread people interrupt when switching to another thread. We find that some subjects are able to vary where they switch depending on how urgent the interrupting task is. When time-allowed, they switched at the end of a discourse segment, which we hypothesize is less disruptive to the interrupted task when it is later resumed.

In this paper, we use computer simulation to better understand mixed-initiative dialogues. We compare two models of mixed-initiative: unrestricted initiative, where either participant can take over control at any point; and restricted initiative where one participant keeps control and the other plays a secondary role, but greater than what single-initiative allows. We find that restricted initiative results in similar solution quality as unrestricted, less communication effort, and similar or less reasoning effort. These results agree with our empirical studies on human-human dialogues, in which we find that participants seem to follow the restricted initiative model.

Mixed-initiative in dialogue does not mean that either speaker is free to take the lead in the conversation at any time. Rather, who can show initiative, and when, is restricted. In this paper, we define initiative and control as two levels of dialogue phenomena, and give further evidence for our theory of restricted initiative that control belongs to the initiator of a discourse segment. We show that the initiator and the non-initiator play different roles in terms of showing initiative.

In this paper, we explore the mechanisms by which conversants control the direction of a dialogue. We find further evidence that control in task-oriented dialogues is subordinate to discourse structure. The initiator of a discourse segment has control; the non-initiator can contribute to the purpose of the segment, but this does not result in that person taking over control. The proposal has important implications for dialogue management, as it will pave the way for building dialogue systems that can engage in mixed initiative dialogues. 1.

Although there are a number of tools for annotating dialogue, little work has been done in visualizing differences among annotations. Visualization of differences in annotation should help in doing consensus annotation, refining an annotation scheme and training novice annotators. In this paper, we present a graphical Annotation Comparison Tool (ACT), which displays multiple annotations side by side and highlights the differences. The differences our tool currently captures are utterance segmentation, utterance order, speech repairs and utterance categories.

The natural language processing component of a speech understanding system is commonly a robust, semantic parser, implemented as either a chart-based transition network, or as a generalized left-right (GLR) parser. In contrast, we are developing a robust, semantic parser that is a single, predictive finite-state machine. Our approach is motivated by our belief that such a finite-state parser can ultimately provide an efficient vehicle for tightly integrating higher-level linguistic knowledge into speech recognition. We report on our development of this parser, with an example of its use, and a description of how it compares to both finite-state predictors and chart-based semantic parsers, while combining the elements of both.

Structured dialogue models are currently the only tools for easily building spoken dialogue systems. This approach, however, requires the dialogue designer to completely specify all dialogue behavior between the user and system, including how information is grounded between the user and the system. In this paper, we advocate factoring out the grounding behavior from structured dialogue models by using a general purpose dialogue manager that accounts for this behavior. This not only simplifies the specification of dialogue, but also allows more powerful mechanisms of grounding to be employed, which cannot be implemented within the framework of structured dialogues. 1. INTRODUCTION As speech recognition and speech synthesis continue to improve, spoken dialogue systems have started to emerge. However, significant barriers remain in building effective spoken dialogue systems. There will always be errors in speech recognition, and unfortunately, natural language understanding and dialogue...

To understand a speaker's turn of a conversation, one needs to segment it into intonational phrases, clean up any speech repairs that might have occurred, and identify discourse markers. In this paper, we argue that these problems must be resolved together, and that they must be resolved early in the processing stream. We put forward a statistical language model that resolves these problem, does POS tagging, and can be used as the language model of a speech recognizer. We find that by accounting for the interactions between these tasks that the performance on each task improves, as does POS tagging and perplexity.

Language models for speech recognition tend to concentrate solely on recognizing the words that were spoken.In this paper, we redefine the speech recognition problem so that its goal is to find both the best sequence of words and their syntactic role (part-of-speech) in the utterance.This is a necessary first step towards tightening the interaction between speech recognition and natural language understanding.

Defining an utterance unit in spoken dialogue has remained a difficult issue.To shed light on this question, we consider grounding behavior in dialogue, and examine co-occurrences between turn-initial grounding acts and utterance unit signals that have been proposed in the literal, namely prosodic boundary tones and pauses.Preliminary results indicate high correlation between grounding and boundary tones, with a secondary correlation for longer pauses.

The TRAINS project is an effort to build a conversationally proficient planning assistant. A key part of the project is the construction of the TRAINS system, which provides the research platform for a wide range of issues in natural language understanding, mixed-initiative planning systems, and representing and reasoning about time, actions and events. Four years have now passed since the beginning of the project. Each year a demonstration system has been produced that focused on a dialogue that illustrates particular aspects of the research. The commitment to building complete integrated systems is a significant overhead on the research, but it is considered essential to guarantee that the results constitute real progress in the field. This paper describes the goals of the project, and the experience with the effort so far.

Interactive spoken dialog provides many new challenges for spoken language systems.One of the most critical is the prevalence of speech repairs.This paper presents an algorithm that detects and corrects speech repairs based on finding the repair pattern.The repair pattern is built by finding word matches and word replacements, and identifying fragments and editing terms.Rather than using a set of prebuilt templates, we build the pattern on the fly.In a fair test, our method, when combined with a statistical model to filter possible repairs, was successful at detecting and correcting 80% of the repairs, without using prosodic information or a parser.

This paper presents a computational model of how conversational participants collaborate in making referring expressions. The model is based on the planning paradigm. It employs plans for constructing and recognizing referring expressions and meta-plans for constructing and recognizing clarifications. This allows the model to account for the generation and understanding both of referring expressions and of their clarifications in a uniform framework using a single knowledge base.