Abstract Automatic dialog systems have become a mainstream part of online customer service. Many such systems are built, maintained, and improved by customer service specialists, rather than dialog systems engineers and computer programmers. As conversations between people and machines become commonplace, it is critical to understand what is working, what is not, and what actions can be taken to reduce the frequency of inappropriate system responses. These analyses and recommendations need to be presented in terms that directly reflect the user experience rather than the internal dialog processing. This paper introduces and explains the use of Actionable Conversational Quality Indicators (ACQIs), which are used both to recognize parts of dialogs that can be improved and to recommend how to improve them. This combines benefits of previous approaches, some of which have focused on producing dialog quality scoring while others have sought to categorize the types of errors the dialog system is making. We demonstrate the effectiveness of using ACQIs on LivePerson internal dialog systems used in commercial customer service applications and on the publicly available LEGOv2 conversational dataset. We report on the annotation and analysis of conversational datasets showing which ACQIs are important to fix in various situations. The annotated datasets are then used to build a predictive model which uses a turn-based vector embedding of the message texts and achieves a 79% weighted average f1-measure at the task of finding the correct ACQI for a given conversation. We predict that if such a model worked perfectly, the range of potential improvement actions a bot-builder must consider at each turn could be reduced by an average of 81%.

Entity linking in dialogue is the task of mapping entity mentions in utterances to a target knowledge base. Prior work on entity linking has mainly focused on well-written articles such as Wikipedia, annotated newswire, or domain-specific datasets. We extend the study of entity linking to open domain dialogue by presenting the OpenEL corpus: an annotated multi-domain corpus for linking entities in natural conversation to Wikidata. Each dialogic utterance in 179 dialogues over 12 topics from the EDINA dataset has been annotated for entities realized by definite referring expressions as well as anaphoric forms such as he, she, it and they. This dataset supports training and evaluation of entity linking in open-domain dialogue, as well as analysis of the effect of using dialogue context and anaphora resolution in model training. It could also be used for fine-tuning a coreference resolution algorithm. To the best of our knowledge, this is the first substantial entity linking corpus publicly available for open-domain dialogue. We also establish baselines for this task using several existing entity linking systems. We found that the Transformer-based system Flair + BLINK has the best performance with a 0.65 F1 score. Our results show that dialogue context is extremely beneficial for entity linking in conversations, with Flair + Blink achieving an F1 of 0.61 without discourse context. These results also demonstrate the remaining performance gap between the baselines and human performance, highlighting the challenges of entity linking in open-domain dialogue, and suggesting many avenues for future research using OpenEL.

There is a substantial population of wheelchair users who do not have the motor capabilities needed to efficiently operate a power wheelchair on their own. Various interfaces have been devised including some simple voice controlled chairs that can understand simple commands. However, such systems are awkward and slow to use. This demonstration shows operation of a smart wheelchair through a spoken conversational interface. By using a more capable dialogue, rather than a simple command paradigm, the chair can leverage off of the user's perceptual capabilities in order to process natural, high-level commands such as take me to the desk, which initiates a conversation with the chair to determine which desk and -if it is not immediately detected by the chair's sensors - where the desk is located.

We present a series of experiments designed to compare grammar-based and robust approaches to speech understanding, performed in the context of an Open Source medical speech translation system. We used two versions of the system, one grammar-based and one robust, trained off the same training data, and evaluated them on test data collected using both versions of the system. The experiments were constructed so as to avoid several methodological problems which occurred in earlier work reported in the literature. We found that the grammarbased version gave significantly better results than the robust version, with the difference increasing as subjects became more familiar with the system's coverage. The rate of improvement in subject performance was positively affected by providing them with an intelligent online help system.

In this paper, we present evidence that providing users of a speech to speech translation system for emergency diagnosis (MedSLT) with a tool that helps them to learn the coverage greatly improves their success in using the system. In MedSLT, the system uses a grammar-based recogniser that provides more predictable results to the translation component. The help module aims at addressing the lack of robustness inherent in this type of approach. It takes as input the result of a robust statistical recogniser that performs better for out-of-coverage data and produces a list of in-coverage example sentences. These examples are selected from a defined list using a heuristic that prioritises sentences maximising the number of N-grams shared with those extracted from the recognition result.

We describe a general side-effect free dialogue management architecture suitable for command and control tasks, which extends the “update semantics” framework by including task as well as dialogue information in the information state. We show that this enables simple and elegant treatments of several dialogue management problems, including corrections, confirmations, querying of the environment, and regression testing. The architecture is discussed in the context of an implemented application, a voice enabled procedure browser for an astronautics domain.

Current alternatives for language modeling are statistical techniques based on large amounts of training data, and hand-crafted context-free or finite-state grammars that are difficult to build and maintain. One way to address the problems of the grammar-based approach is to compile recognition grammars from grammars written in a more expressive formalism. While theoretically straight-forward, the compilation process can exceed memory and time bounds, and might not always result in accurate and efficient speech recognition. We will describe and evaluate two approaches to this compilation problem. We will also describe and evaluate additional techniques to reduce the structural ambiguity of the language model.

This paper presents a minimal enumerative approach to the problem of compiling typed unification grammars into CFG language models, a prototype implementation and results of experiments in which it was used to compile some non-trivial unification grammars. We argue that enumerative methods are considerably more useful than has been previously believed. Also, the simplicity of enumerative methods makes them a natural baseline against which to compare alternative approaches. 1.

We analyze what functions as a YES response and a NO response for different yes/no questions. This problem is surprisingly complex: respondents do not always produce overt yes or no lexical items in response to a yes/no question. In addition, when respondents don't include a clear yes or no word, they may mean to communicate a clear YES or NO meaning, or something else. We find that the classification of yes/no questions described in (Carletta et al., 1995) for the Edinburgh map task corpus correlates well with whether a response will be a bare yes or no, a yes or no plus additional speech, or just speech without an overt yes or no. Correlation with responses described simply as as "direct" or "indirect" is less good. We also find that, under the three-way categorization, the strength of a question's expectation for a YES response predicts the form of the response. 1 Introduction In order to model dialogue, it is necessary to understand what kinds of utterances are likely to occur in...

This paper addresses the problem of accented occurrences of the verb believe that cannot be accounted for as instances of focus by either semantic or pragmatic accounts. In the natural interpretation of these accented believe utterances, the embedded sentence expresses the main proposition while the believe serves to reduce the speaker's accountability. I present an account of these problematic utterances which captures the accountability aspect of the interpretation and explains both similarities and differences between these cases and other instances of focus. Speaker accountability is modeled by ease of abandonment of a proposition within a belief revision system. I propose that the function of accented believe is to explicitly mark a proposition as more easily abandoned than the contextually determined default. Differences between accented believe and other instances of focus are accounted for by believe being external to the main proposition. 1 INTRODUCTION Sentences such as (...