In social interactions between humans and Embodied Conversational Agents (ECAs) conversational interruptions may occur. ECAs should be prepared to detect, manage and react to such interruptions in order to keep the interaction smooth, natural and believable. In this paper, we examined nonverbal reactions exhibited by an interruptee during conversational interruptions and we propose a novel technique driven by an evolutionary algorithm to build a computational model for ECAs to manage user's interruptions. We propose a taxonomy of conversational interruptions adapted from social psychology, an annotation schema for semi-automatic detection of user's interruptions and a corpus-based observational analysis of human nonverbal reactions to interruptions. Then we present a methodology for building an ECA behavioral model including the design and realization of an interactive study driven by an evolutionary algorithm, where participants interactively built the most appropriate set of multimodal reactive behaviours for an ECA to display interpersonal attitudes (friendly/hostile) through nonverbal reactions to a conversational interruption.

This paper presents a computational model for managing an Embodied Conversational Agent's first impressions of warmth and competence towards the user. These impressions are important to manage because they can impact users' perception of the agent and their willingness to continue the interaction with the agent. The model aims at detecting user's impression of the agent and producing appropriate agent's verbal and nonverbal behaviours in order to maintain a positive impression of warmth and competence. User's impressions are recognized using a machine learning approach with facial expressions (action units) which are important indicators of users' affective states and intentions. The agent adapts in real-time its verbal and nonverbal behaviour, with a reinforcement learning algorithm that takes user's impressions as reward to select the most appropriate combination of verbal and non-verbal behaviour to perform. A user study to test the model in a contextualized interaction with users is also presented. Our hypotheses are that users' ratings differs when the agents adapts its behaviour according to our reinforcement learning algorithm, compared to when the agent does not adapt its behaviour to user's reactions (i.e., when it randomly selects its behaviours). The study shows a general tendency for the agent to perform better when using our model than in the random condition. Significant results shows that user's ratings about agent's warmth are influenced by their a-priori about virtual characters, as well as that users' judged the agent as more competent when it adapted its behaviour compared to random condition.

In this paper we present an analysis from a corpus of dyadic expert-novice knowledge sharing interactions. The analysis aims at investigating the relationship between observed non-verbal cues and first impressions formation of warmth and competence. We first obtained both discrete and continuous annotations of our data. Discrete descriptors include non-verbal cues such as type of gestures, arms rest poses, head movements and smiles. Continuous descriptors concern annotators' judgments of the expert's warmth and competence during the observed interaction with the novice. Then we computed Odds Ratios between those descriptors. Results highlight the role of smiling in warmth and competence impressions. Smiling is associated with increased levels of warmth and decreasing competence. It also affects the impact of others non-verbal cues (e.g. self-adaptors gestures) on warmth and competence. Moreover, our findings provide interesting insights about the role of rest poses, that are associated with decreased levels of warmth and competence impressions.

We present a novel multi-lingual database of natural dyadic novice-expert interactions, named NoXi, featuring screen-mediated dyadic human interactions in the context of information exchange and retrieval. NoXi is designed to provide spontaneous interactions with emphasis on adaptive behaviors and unexpected situations (e.g. conversational interruptions). A rich set of audio-visual data, as well as continuous and discrete annotations are publicly available through a web interface. Descriptors include low level social signals (e.g. gestures, smiles), functional descriptors (e.g. turn-taking, dialogue acts) and interaction descriptors (e.g. engagement, interest, and fluidity).

We present a demonstration of the ARIA framework, a modular approach for rapid development of virtual humans for information retrieval that have linguistic, emotional, and social skills and a strong personality. We demonstrate the framework's capabilities in a scenario where `Alice in Wonderland', a popular English literature book, is embodied by a virtual human representing Alice. The user can engage in an information exchange dialogue, where Alice acts as the expert on the book, and the user as an interested novice. Besides speech recognition, sophisticated audio-visual behaviour analysis is used to inform the core agent dialogue module about the user's state and intentions, so that it can go beyond simple chat-bot dialogue. The behaviour generation module features a unique new capability of being able to deal gracefully with interruptions of the agent.

Virtual Reality and immersive experiences, which allow players to share the same virtual environment as the characters of a virtual world, have gained more and more interest recently. In order to conceive these immersive virtual worlds, one of the challenges is to give to the characters that populate them the ability to express behaviors that can support the immersion. In this work, we propose a model capable of controlling and simulating a conversational group of social agents in an immersive environment. We describe this model which has been previously validated using a regular screen setting and we present a study for measuring whether users recognized the attitudes expressed by virtual agents through the realtime generated animations of nonverbal behavior in an immersive setting. Results mirrored those of the regular screen setting thus providing further insights for improving players experiences by integrating them into immersive simulated group conversations with characters that express different interpersonal attitudes.

In the everyday world people form small conversing groups where social interaction takes place, and much of the social behavior takes place through managing interpersonal space (i.e., proxemics) and group formation, signaling their attentio to others (i.e., through gaze behavior), and expressing certain attitudes, for example, friendliness, by smiling, getting close through increased engagement and intimacy, and welcoming newcomers. Many real-time interactive systems feature virtual anthropomorphic characters in order to simulate conversing groups and add plausibility and believability to the simulated environments. However, only a few have dealt with autonomous behavior generation, and in those cases, the agents’ exhibited behavior should be evaluated by users in terms of appropriateness, believability, and conveyed meaning (e.g., attitudes). In this article we present an integrated intelligent interactive system for generating believable nonverbal behavior exhibited by virtual agents in small simulated group conversations. The produced behavior supports group formation management and the expression of interpersonal attitudes (friendly vs. unfriendly) both among the agents in the group (i.e., in-group attitude) and towards an approaching user in an avatar-based interaction (out-group attitude). A user study investigating the effects of these attitudes on users’ social presence evaluation and proxemics behavior (with their avatar) in a three-dimensional virtual city environment is presented. We divided the study into two trials according to the task assigned to users, that is, joining a conversing group and reaching a target destination behind the group. Results showed that the out-group attitude had a major impact on social presence evaluations in both trials, whereby friendly groups were perceived as more socially rich. The user’s proxemics behavior depended on both out-group and in-group attitudes expressed by the agents. Implications of these results for the design and implementation of similar intelligent interactive systems for the autonomous generation of agents’ multimodal behavior are briefly discussed.

In greeting encounters, first impressions of personality and attitude are quickly formed and might determine important relational decisions, such as the likelihood and frequency of subsequent encounters. An anthropomorphic user interface is not immune to these judgments, specifically when exhibiting social interaction skills in public spaces. A favorable impression may help engaging users in interaction and attaining acceptance for long-term interactions. We present three studies implementing a model of first impressions for initiating user interactions with an anthropomorphic museum guide agent with socio-relational skills. We focus on nonverbal behavior exhibiting personality and interpersonal attitude. In two laboratory studies, we demonstrate that impressions of an agent's personality are quickly formed based on proximity, whereas interpersonal attitude is conveyed through smile and gaze. We also found that interpersonal attitude has greater impact than personality on the user's decision to spend time with the agent. These findings are then applied to a museum guide agent exhibited at the Boston Museum of Science. In this field study, we show that employing our model increases the number of visitors engaging in interaction.