Combining conversational AI with refreshable tactile displays (RTDs) offers significant potential for creating accessible data visualization for people who are blind or have low vision (BLV). To support researchers and developers building accessible data visualizations with RTDs, we present a multimodal data interaction architecture along with an open-source reference implementation. Our system is the first to combine touch input with a conversational agent on an RTD, enabling deictic queries that fuse touch context with spoken language, such as "what is the trend between these points?" The architecture addresses key technical challenges, including touch sensing on RTDs, visual-to-tactile encoding, integrating touch context with conversational AI, and synchronizing multimodal output. Our contributions are twofold: (1) a technical architecture integrating RTD hardware, external touch sensing, and conversational AI to enable multimodal data interaction; and (2) an open-source reference implementation demonstrating its feasibility. This work provides a technical foundation to support future research in multimodal accessible data visualization.

In this paper, we generate two types of explanations, Impact and Critique, of predictions made by a Goal Recognizer (GR) – a system that infers agents’ goals from observations. Impact explanations describe the top-predicted goal(s) and the main observations that led to these predictions. Critique explanations augment these explanations with evidence that challenges the GR’s predictions if so warranted. Our user study compares users’ goal-recognition accuracy for Impact and Critique explanations, and users’ views about these explanations, under three prediction-correctness conditions: correct, partially correct and incorrect. Our results show that (1) users stick with a GR’s predictions, even when a Critique explanation highlights its flaws; yet (2) Critique explanations are deemed better than Impact explanations in most respects.

Despite the recent surge of research efforts to make data visualizations accessible to people who are blind or have low vision (BLV), how to support BLV people's data analysis remains an important and challenging question. As refreshable tactile displays (RTDs) become cheaper and conversational agents continue to improve, their combination provides a promising approach to support BLV people's interactive data exploration and analysis. To understand how BLV people would use and react to a system combining an RTD with a conversational agent, we conducted a Wizard-of-Oz study with 11 BLV participants, where they interacted with line charts, bar charts, and isarithmic maps. Our analysis of participants' interactions led to the identification of nine distinct patterns. We also learned that the choice of modalities depended on the type of task and prior experience with tactile graphics, and that participants strongly preferred the combination of RTD and speech to a single modality. In addition, participants with more tactile experience described how tactile images facilitated a deeper engagement with the data and supported independent interpretation. Our findings will inform the design of interfaces for such interactive mixed-modality systems.

In this paper, we generate and compare three types of explanations of Machine Learning (ML) predictions: simple, conservative and unifying.Simple explanations are concise, conservative explanations address the surprisingness of a prediction, and unifying explanations convey the extent to which an ML model's predictions are applicable.The results of our user study show that (1) conservative and unifying explanations are liked equally and considered largely equivalent in terms of completeness, helpfulness for understanding the AI, and enticement to act, and both are deemed better than simple explanations; and (2) users' views about explanations are influenced by the (dis)agreement between the ML model's predictions and users' estimations of these predictions, and by the inclusion/omission of features users expect to see in explanations.

We consider two types of numeric representations for conveying the uncertainty of predictions made by Machine Learning (ML) models: confidence-based (e.g., "the AI is 90% confident") and frequency-based (e.g., "the AI was correct in 180 (90%) out of 200 cases").We conducted a user study to determine which factors influence users' acceptance of predictions made by ML models, and how the two types of uncertainty representations affect users' views about explanations.Our results show that users' acceptance of ML model predictions depends mainly on the models' confidence, and that explanations that include uncertainty information are deemed better in several respects than explanations that omit it, with frequency-based representations being deemed better than confidencebased representations.

Building a dialogue agent that can seamlessly interact with humans, in multi-modal regimes, requires two fundamental abilities: (1) understanding emotion and dialogue acts within situated user scenarios, and (2) grounding perceived visual cues to dialogue contexts.However, recent works have uncovered shortcomings of existing dialogue agents in understanding emotions and dialogue acts, and in grounding visual cues effectively.In this work, we investigate whether additional dialogue data with only visual descriptions can help dialogue agents effectively align visual and textual features, and enhance the ability of dialogue agents to ground perceived visual cues to dialogue contexts.To this end, in the absence of a suitable dataset, we propose a synthetic visual description generation pipeline, and contribute a large-scale synthetic visual description dataset.In addition, we propose a general training procedure for effectively leveraging these synthetic data.We conduct comprehensive analyses to evaluate the impact of synthetic data on two benchmarks: MELD and IEMO-CAP.Our findings suggest that synthetic visual descriptions can serve as an effective way to enhance a dialogue agents' grounding ability, and that the training scheme affects the extent to which these descriptions improve the agent's performance.

Content Warning: this paper may contain content that is offensive or upsetting.

We consider the influence of two types of contextual information, background information available to users and users’ goals , on users’ views and preferences regarding textual explanations generated for the outcomes predicted by Decision Trees (DTs). To investigate the influence of background information, we generate contrastive explanations that address potential conflicts between aspects of DT predictions and plausible expectations licensed by background information. We define four types of conflicts, operationalize their identification, and specify explanatory schemas that address them. To investigate the influence of users’ goals, we employ an interactive setting where given a goal and an initial explanation for a predicted outcome, users select follow-up questions, and assess the explanations that answer these questions. Here, we offer algorithms to generate explanations that address six types of follow-up questions. The main result from both user studies is that explanations which have a contrastive aspect about a predicted class are generally preferred by users. In addition, the results from the first study indicate that these explanations are deemed especially valuable when users expectations differ from predicted outcomes; and the results from the second study indicate that contrastive explanations which describe how to change a predicted outcome are particularly well regarded in terms of helping users achieve this goal, and they are also popular in terms of helping users achieve other goals.

Abstract Monitoring systems have become increasingly prevalent in order to increase the safety of elderly people who live alone. These systems are designed to raise alerts when adverse events are detected, which in turn enables family and carers to take action in a timely manner. However, monitoring systems typically suffer from two problems: they may generate false alerts or miss true adverse events. This motivates the two user studies presented in this paper: (1) in the first study, we investigate the effect of the performance of different monitoring systems, in terms of accuracy and error type, on users’ trust in these systems and behaviour; and (2) in the second study, we examine the effect of recommendations made by an advisor agent on users’ behaviour. Our user studies take the form of a web-based game set in a retirement village, where elderly residents live in smart homes equipped with monitoring systems. Players, who “work” in the village, perform a primary task whereby they must ensure the welfare of the residents by attending to adverse events in a timely manner, and a secondary routine task that demands their attention. These conditions are typical of a retirement setting, where workers perform various duties in addition to keeping an eye on a monitoring system. Our main findings pertain to: (1) the identification of user types that shed light on users’ trust in automation and aspects of their behaviour; (2) the effect of monitoring-system accuracy and error type on users’ trust and behaviour; (3) the effect of the recommendations made by an advisor agent on users’ behaviour; and (4) the identification of influential factors in models that predict users’ trust and behaviour. The studies that yield these findings are enabled by two methodological contributions: (5) the game itself, which supports experimentation with various factors, and a version of the game augmented with an advisor agent; and (6) techniques for calibrating the parameters of the game and determining the recommendations of the advisor agent.

Xuelin Situ, Ingrid Zukerman, Cecile Paris, Sameen Maruf, Gholamreza Haffari. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Lifelong Learning (LL) black-box models are dynamic in that they keep learning from new tasks and constantly update their parameters.Owing to the need to utilize information from previously seen tasks, and capture commonalities in potentially diverse data, it is hard for automatic explanation methods to explain the outcomes of these models.In addition, existing explanation methods, e.g., LIME (Ribeiro et al., 2016), which are computationally expensive when explaining a static black-box model, are even more inefficient in the LL setting.In this paper, we propose a novel Lifelong Explanation (LLE) approach that continuously trains a student explainer under the supervision of a teacher -an arbitrary explanation algorithm -on different tasks undertaken in LL.We also leverage the Experience Replay (ER) mechanism to prevent catastrophic forgetting in the student explainer.Our experiments comparing LLE to three baselines on text classification tasks show that LLE can enhance the stability of the explanations for all seen tasks and maintain the same level of faithfulness to the black-box model as the teacher, while being up to 10 2 times faster at test time.Our ablation study shows that the ER mechanism in our LLE approach enhances the learning capabilities of the student explainer.Our code is available at https://github.com/situsnow/LLE.

We offer an approach to explain Decision Tree (DT) predictions by addressing potential conflicts between aspects of these predictions and plausible expectations licensed by background information.We define four types of conflicts, operationalize their identification, and specify explanatory schemas that address them.Our human evaluation focused on the effect of explanations on users' understanding of a DT's reasoning and their willingness to act on its predictions.The results show that (1) explanations that address potential conflicts are considered at least as good as baseline explanations that just follow a DT path; and (2) the conflictbased explanations are deemed especially valuable when users' expectations disagree with the DT's predictions.

We describe a longitudinal user study conducted in the context of a Spoken Dialogue System for a household robot, where we examined the influence of time displacement and situational risk on users’ preferred responses. To this effect, we employed a corpus of spoken requests that asked a robot to fetch or move objects in a room. In the first stage of our study, participants selected among four response types to these requests under two risk conditions: low and high. After some time, the same participants rated several responses to the previous requests — these responses were instantiated from the four response types. Our results show that participants did not rate highly their own response types; moreover, they rated their own response types similarly to different ones. This suggests that, at least in this context, people’s preferences at a particular point in time may not reflect their general attitudes, and that various reasonable response types may be equally acceptable. Our study also reveals that situational risk influences the acceptability of some response types.

We present insights obtained from a web-based game designed to investigate trust-related factors in a care-taking scenario. The game is set in a retirement village, where elderly residents live in smart homes equipped with monitoring systems. These systems should raise alerts when adverse events happen, but they do not function perfectly (they may issue false alerts or miss true events). Players, who "work'' in the village, perform a primary task, whereby they must ensure the welfare of the residents by attending to adverse events in a timely manner, and a secondary routine task that demands their attention. Our contributions are (1) the game itself, which supports experimentation with various trust-related factors; (2) a methodology for the calibration of the game's parameters; (3) insights from two experiments regarding the relationship between device performance, in particular error type, and trust and user behaviour; and (4) insights from predictive models about factors that influence trust and aspects of user behaviour.

Quan Hung Tran, Tuan Lai, Gholamreza Haffari, Ingrid Zukerman, Trung Bui, Hung Bui. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). 2018.

We propose a novel hierarchical Recurrent Neural Network (RNN) for learning sequences of Dialogue Acts (DAs).The input in this task is a sequence of utterances (i.e., conversational contributions) comprising a sequence of tokens, and the output is a sequence of DA labels (one label per utterance).Our model leverages the hierarchical nature of dialogue data by using two nested RNNs that capture long-range dependencies at the dialogue level and the utterance level.This model is combined with an attention mechanism that focuses on salient tokens in utterances.Our experimental results show that our model outperforms strong baselines on two popular datasets, Switchboard and MapTask; and our detailed empirical analysis highlights the impact of each aspect of our model.

This paper introduces a novel training/decoding strategy for sequence labeling.Instead of greedily choosing a label at each time step, and using it for the next prediction, we retain the probability distribution over the current label, and pass this distribution to the next prediction.This approach allows us to avoid the effect of label bias and error propagation in sequence learning/decoding.Our experiments on dialogue act classification demonstrate the effectiveness of this approach.Even though our underlying neural network model is relatively simple, it outperforms more complex neural models, achieving state-of-the-art results on the MapTask and Switchboard corpora.

Quan Hung Tran, Ingrid Zukerman, Gholamreza Haffari. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2016.

This paper presents a study to determine users' attitudes toward smart devices. We conducted a web survey to elicit users' ratings for devices and combinations of tasks and devices; the results of this survey led to the development of a Recommender System (RS) for smart devices for particular tasks. We investigated user- and item-based Collaborative Filters, and compared their performance with that of global and demographic RS baselines. We then developed a technique based on Principal Components Analysis to select a subset of the original survey questions that supports the prediction of users' ratings for device-task combinations. Our results show that the accuracy of an RS that asks only a small subset of the survey questions is similar to that of an RS that predicts users' answers to one survey question on the basis of their answers to all the other questions.

Authorship attribution deals with identifying the authors of anonymous texts. Traditionally, research in this field has focused on formal texts, such as essays and novels, but recently more attention has been given to texts generated by on-line users, such as e-mails and blogs. Authorship attribution of such on-line texts is a more challenging task than traditional authorship attribution, because such texts tend to be short, and the number of candidate authors is often larger than in traditional settings. We address this challenge by using topic models to obtain author representations. In addition to exploring novel ways of applying two popular topic models to this task, we test our new model that projects authors and documents to two disjoint topic spaces. Utilizing our model in authorship attribution yields state-of-the-art performance on several data sets, containing either formal texts written by a few authors or informal texts generated by tens to thousands of on-line users. We also present experimental results that demonstrate the applicability of topical author representations to two other problems: inferring the sentiment polarity of texts, and predicting the ratings that users would give to items such as movies.

OBJECTIVE: To develop a conceptual framework for the design of an in-home monitoring system (IMS) based on the requirements of older adults with vision impairment (VI), informal caregivers and eye-care rehabilitation professionals. MATERIALS AND METHODS: Concept mapping, a mixed-methods statistical research tool, was used in the construction of the framework. Overall, 40 participants brainstormed or sorted and rated 83 statements concerning an IMS for older adults with VI. Multidimensional scaling and hierarchical cluster analysis were employed to construct the framework. A questionnaire yielded further insights into the views of a wider sample of older adults with VI (n=78) and caregivers (n=25) regarding IMS. RESULTS: Concept mapping revealed a nine-cluster model of IMS-related aspects including affordability, awareness of system capabilities, simplicity of installation, operation and maintenance, system integrity and reliability, fall detection and safe movement, user customization, user preferences regarding information delivery, and safety alerts for patients and caregivers. From the questionnaire, independence, safety and fall detection were the most commonly reported reasons for older adults and caregivers to accept an IMS. Concerns included cost, privacy, security of the information obtained through monitoring, system accuracy, and ease of use. DISCUSSION: Older adults with VI, caregivers and professionals are receptive to in-home monitoring, mainly for fall detection and safety monitoring, but have concerns that must be addressed when developing an IMS. CONCLUSION: Our study provides a novel conceptual framework for the design of an IMS that will be maximally acceptable and beneficial to our ageing and vision-impaired population.

We present a probabilistic, salience-based mechanism for the interpretation of pointing gestures together with spoken utterances. Our formulation models dependencies between spatial and temporal aspects of gestures and features of objects. The results from our corpus-based evaluation show that the incorporation of pointing information improves interpretation accuracy. Index Terms: understanding spoken language and gesture, probabilistic approach 1.

This article presents an investigation of corpus-based methods for the automation of help-desk e-mail responses. Specifically, we investigate this problem along two operational dimensions: (1) information-gathering technique, and (2) granularity of the information. We consider two information-gathering techniques (retrieval and prediction) applied to information represented at two levels of granularity (document-level and sentence-level). Document-level methods correspond to the reuse of an existing response e-mail to address new requests. Sentence-level methods correspond to applying extractive multi-document summarization techniques to collate units of information from more than one e-mail. Evaluation of the performance of the different methods shows that in combination they are able to successfully automate the generation of responses for a substantial portion of e-mail requests in our corpus. We also investigate a meta-selection process that learns to choose one method to address a new inquiry e-mail, thus providing a unified response automation solution.

A Spoken Language Interpretation Component for a Robot Dialogue System

This paper presents a corpus study that explores the extent to which captions contribute to recognizing the intended message of an information graphic. It then presents an implemented graphic interpretation system that takes into account a variety of communicative signals, and an evaluation study showing that evidence obtained from shallow processing of the graphic's caption has a significant impact on the system's success. This work is part of a larger project whose goal is to provide sight-impaired users with effective access to information graphics.

Argumentation may be defined as the study of different aspects of (human) interactions whose objective is to reach a conclusion about the truth of a proposition or the adoption of a course of action.The aspects in question include the argumentation interaction, which is typically a dialogue; the construction and presentation of arguments for or against a proposition; the interpretation of arguments; the knowledge representation required to build and interpret arguments; theory of rhetoric; and applications such as agent negotiation, legal argument, and decision-support systems.Argumentation is a rich interdisciplinary area of study, which so far has largely been considered separately by practitioners in different fields.To address this problem, Reed and Norman brought together researchers in argumentation, artificial intelligence, law, linguistics, and philosophy, with the aim of writing a book about argumentation that would foster interdisciplinary work (see the book's Preface).The book comprises six chapters, each written by a different group of people, which cover diverse areas where argumentation has computational implications.Chapter 1 introduces the book.Chapters 2-4 consider argumentation in the context of multiagent, decision-support, and legal systems, focusing on rhetorical and representational issues that need to be taken into account when implementing such systems.Chapters 5 and 6 address computational aspects of the implementation of argumentation systems.As indicated in its Preface, the book distances itself from logical approaches to argumentation, and instead focuses on natural language interactions and conversational scenarios.The six chapters of the book are outlined below.''A Roadmap of Research in Argument and Computation,'' by Reed and Norman, motivates the subsequent chapters and provides background on three main areas of research that are relevant to these chapters: artificial intelligence, rhetoric, and multiagent systems.The discussion is supported by an extensive list of references (although there are some omissions).In ''Argument and Multi-agent Systems,'' by Norman, Carbogim, Krabbe, and Walton, the authors address issues at the intersection of argumentation and multiagent systems.The discussion starts by contrasting argumentation theory with agent communication along two main dimensions: dialogue-structure/interaction-protocols, and beliefs/commitments.The chapter then proceeds to describe different types of dialogue, argument schemas (Walton 1996), and models of agent dialogue.A recurrent

We investigate two seemingly incompatible approaches for improving document retrieval performance in the context of question answering: query expansion and query reduction. Queries are expanded by generating lexical paraphrases. Syntactic, semantic and corpus-based frequency information is used in this process. Queries are reduced by removing words that may detract from retrieval performance. Features that identify these words were obtained from decision graphs. These approaches were evaluated using a subset of queries from TREC8, 9 and 10. Our evaluation shows that each approach in isolation improves retrieval performance, and both approaches together yield substantial improvements. Specifically, query expansion followed by reduction improved the average number of correct documents retrieved by 21.7% and the average number of queries that can be answered by 15%.

The explicit consideration of an addressee's inferences during discourse planning affects the content and coherence of the generated discourse. The content of the discourse is affected because the consideration of an addressee's inferences may require the addition of information that addresses erroneous inferences or suggest the omission of easily inferred information. The coherence of the discourse is affected because the generated discourse should incorporate inferential relations that emerge from the consideration of an addressee's inferences. In this article, we describe a discourse planner that takes into consideration a user's inferences. In particular, we discuss its content planning and discourse structuring mechanisms. The content planning mechanism generates a set of rhetorical devices that achieves a given communicative goal. The discourse structuring mechanism takes into consideration schematic, inferential, and prescriptive relations between these rhetorical devices in order to organize them into a coherent sequence. Our discourse planner has been implemented in a system called WISHFUL that generates explanations about concepts in technical domains. We evaluated WISHFUL's performance to test the effect of considering a user's inferences during content planning and discourse organization and also to determine the suitability of the discourse produced by WISHFUL for different types of users. The results of our evaluation endorse the ideas embodied in the WISHFUL system.

We describe a mechanism which generates rebuttals to a user's rejoinders in the context of arguments generated from Bayesian networks. This mechanism is implemented in an interactive argumentation system. Given an argument generated by the system and an interpretation of a user's rejoinder, the generation of the rebuttal takes into account the intended effect of the user's rejoinder, determined on a model of the user's beliefs, and its actual effect, determined on a model of the system's beliefs. We consider three main rebuttal strategies: refute the user's rejoinder, strengthen the argument goal, and dismiss the user's line of reasoning.

During argumentation, people persuade their audience using a variety of strategies, e.g., hypothetical reasoning, reasoning by cases and ordinary premise-to-goal arguments. In this paper, we offer an operational definition of the conditions for pursuing these strategies, and incorporate into a Bayesian argument-generation system a mechanism for proposing applicable argumentation strategies, generating specific arguments based on these strategies, and selecting a final argument.

We explore the problem of how to perform complex strategic planning in an uncertain domain. Although card games such as bridge and poker provide interesting domains for exploring uncertainty, they tend to be strategically simple given perfect information. We present and discuss a strategically complex, easily controlled, imperfect information domain-a variant of chess-which allows us to explore this difficult problem. We also describe a modular hybrid architecture that uses game theory, information theory and utility theory to play this new game. We discuss preliminary results from our architecture and consider possible future directions for the project.

Human discourse is fraught with rhetorical devices such as contradictions, illustrations, and analogies. These rhetorical devices carry important information which a listener uses to speed up the comprehension process. In this paper, we present a taxonomy of rhetorical devices commonly used in tutoring environment, and model the meaning of a class of rhetorical devices in terms of their anticipated effect on a listener's knowledge. These predictions are then used in planning computer‐generated discourse. As a testbed for our ideas, a system called WISHFUL is being implemented to generate commentaries in the domain of high‐school algebra within the framework of an intelligent tutoring system.