For affective computing to have an impact outside the laboratory, facial expressions must be studied in rich naturalistic situations. We argue negotiations are one such situation as they are ubiquitous in daily life, often evoke strong emotions, and perceived emotion shapes decisions and outcomes. Negotiations are a growing focus in AI research and applications, including agents that negotiate directly with people and attempt to use affective information. We introduce the DyNego-WOZ Corpus, which includes dyadic negotiation between participants and wizard-controlled virtual humans. We demonstrate the value of this corpus to the affective computing community by examining participants’ facial expressions in response to a virtual human negotiation partner. We show that people's facial expressions typically co-occur with the end of their partner's speech (suggesting they reflect a reaction to the content of this speech), that these reactions do not correspond to prototypicalemotionalexpressions, and that these reactions can help predict the expresser's subsequent action. We highlight challenges in working with such naturalistic data, including difficulties of expression recognition during speech, and the extreme variability of expressions, both across participants and within a negotiation. Our findings reinforce arguments that facial expressions convey more than emotional state but serve important communicative functions.

Women earn less than men in technical fields. Competing theories have been offered to explain this disparity. Some argue that women underperform in negotiating their salary, in-part due to language in job descriptions, called gender triggers, which leave women feeling disadvantaged in salary negotiations. Others point to structural and institutional bias: i.e., recruiters make better offers to men even when women exhibit equal negotiation skills. As a final salary is co-constructed though an interaction between employees and recruiters, it is difficult to disentangle these views. Here, we discuss how intelligent virtual agents serve as powerful methodological tools that lend new insight into this psychological debate. We use virtual negotiators to examine the impact of gender triggers on computer science (CS) undergraduates that engaged in a simulated salary negotiation with an automated recruiter. We find that, regardless of gender, CS students are reluctant to negotiate, and this hesitancy likely lowers their starting salary. Even when they negotiate, students show little skill in discovering tradeoffs that could enhance their salary, highlighting the need for negotiation training in technical fields. Most importantly, we find little evidence that gender triggers impact women's negotiated outcomes, at least within the field of CS. We argue that findings that emphasize women's individual deficits may reflect a lack of experimental control, which intelligent agents can help correct, and that structural and institutional explanations of inequity deserve greater attention.

We apply Reinforcement Learning (RL) to the problem of incremental dialogue policy learning in the context of a fast-paced dialogue game.We compare the policy learned by RL with a high performance baseline policy which has been shown to perform very efficiently (nearly as well as humans) in this dialogue game.The RL policy outperforms the baseline policy in offline simulations (based on real user data).We provide a detailed comparison of the RL policy and the baseline policy, including information about how much effort and time it took to develop each one of them.We also highlight the cases where the RL policy performs better, and show that understanding the RL policy can provide valuable insights which can inform the creation of an even better rule-based policy.

Real-world scenes typically have complex structure, and utterances about them consequently do as well.We devise and evaluate a model that processes descriptions of complex configurations of geometric shapes and can identify the described scenes among a set of candidates, including similar distractors.The model works with raw images of scenes, and by design can work word-by-word incrementally.Hence, it can be used in highly-responsive interactive and situated settings.Using a corpus of descriptions from game-play between human subjects (who found this to be a challenging task), we show that reconstruction of description structure in our system contributes to task success and supports the performance of the word-based model of grounded semantics that we use.

In this paper, we present and evaluate an approach to incremental dialogue act (DA) segmentation and classification.Our approach utilizes prosodic, lexico-syntactic and contextual features, and achieves an encouraging level of performance in offline corpus-based evaluation as well as in simulated human-agent dialogues.Our approach uses a pipeline of sequential processing steps, and we investigate the contribution of different processing steps to DA segmentation errors.We present our results using both existing and new metrics for DA segmentation.The incremental DA segmentation capability described here may help future systems to allow more natural speech from users and enable more natural patterns of interaction.

We present the SimSensei system, a fully automatic virtual agent that conducts interviews to assess indicators of psychological distress. We emphasize on the perception part of the system, a multimodal framework which captures and analyzes user state for both behavioral understanding and interactional purposes.

We present the SimSensei system, a fully automatic virtual agent that conducts interviews to assess indicators of psychological distress. With this demo, we focus our attention on the perception part of the system, a multimodal framework which captures and analyzes user state behavior for both behavioral understanding and interactional purposes. We will demonstrate real-time user state sensing as a part of the SimSensei architecture and discuss how this technology enabled automatic analysis of behaviors related to psychological distress.

This paper introduces Eve, a highperformance agent that plays a fast-paced image matching game in a spoken dialogue with a human partner.The agent can be optimized and operated in three different modes of incremental speech processing that optionally include incremental speech recognition, language understanding, and dialogue policies.We present our framework for training and evaluating the agent's dialogue policies.In a user study involving 125 human participants, we evaluate three incremental architectures against each other and also compare their performance to human-human gameplay.Our study reveals that the most fully incremental agent achieves game scores that are comparable to those achieved in human-human gameplay, are higher than those achieved by partially and nonincremental versions, and are accompanied by improved user perceptions of efficiency, understanding of speech, and naturalness of interaction.

Fabrizio Morbini, David DeVault, Kallirroi Georgila, Ron Artstein, David Traum, Louis-Philippe Morency. Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL). 2014.

We explore the potential for a responsive spoken dialogue system to use the real-time status of an incremental speech understanding model to guide its incremental decision-making about how to respond to a user utterance that is still in progress. Spoken dialogue systems have a range of potentially useful realtime response options as a user is speaking, such as providing acknowledgments or backchannels, interrupting the user to ask a clarification question or to initiate the system’s response, or even completing the user’s utterance at appropriate moments. However, implementing such incremental response capabilities seems to require that a system be able to assess its own level of understanding incrementally, so that an appropriate response can be selected at each moment. In this paper, we use a datadriven classification approach to explore the trade-offs that a virtual human dialogue system faces in reliably identifying how its understanding is progressing during a user utterance.

From Domain Specification to Virtual Humans : An integrated approach to authoring tactical questioning characters

We describe a practical method for measuring eye movements during psychological tests. This is an important class of applications including clinical evaluations and marketing studies. Existing methods in common use for psychological measurement, for example infrared reflection methods, are invasive involving head stabilization and special purpose lighting. In our experiments, we need to observe subjects for long periods, on the order of one hour. In addition, subjects must verbalize, which makes it difficult to stabilize their heads relative to the camera. We track the head using a lightweight spectacle framework worn by the subject. It has a set of easily visible colored balls. We segment each image into four characteristic colors, corresponding to iris, yellow ball, red ball, and background, which are obtained by sampling the images for each subject. The classification into colors is done by training a simple neural network for each characteristic color. We match a template to color-reduced image regions to find the balls and the two irises. We use a model-based object pose method, which uses a prior measurement of the relative positions of the balls, to calculate the spectacle framework pose (the head pose). A linear method is used for calibrating gaze position against head pose and iris positions. The subject's gaze position can be traded reliably for periods of more than an hour. The locations of image features are found with an accuracy of approximately one pixel of the image. In a 640/spl times/480 image of the whole face, the eyes are each about 80 pixels across. This gives a corresponding accuracy of calculated eye gaze position on a 17 inch monitor of about 1 cm horizontally and 2 cm vertically. This method has shown itself in practice to be very flexible for psychological measurement, giving sufficient accuracy and being noninvasive.