In fields like military aviation, virtual assistance is crucial to manage the overwhelming amount of information pilots deal with, reducing their workload. This study seeks to determine the key factors for effective collaboration between pilots and assistants, focusing on embodiment and communication style, which may vary based on the situation criticality. A pilot evaluation show that both embodiment and communication style should be adapted to operational context, especially critical ones (i.e. presence of VA after a loss of consciousness and task oriented communication style during combat phases).

To improve communication and collaboration in virtual reality (VR), we suggest going beyond improving behavioral realism on participant avatars. Leveraging VR capabilities, our approach focuses on transforming and amplifying social interactions. VR technology prevents users from naturally expressing social signals, accurately detecting and transmitting them due to still limited technological capabilities, or even fully perceiving them. We propose an augmented model aimed at enhancing collaboration by displaying non-verbal engagement behaviors on listeners’ avatars, even if not executed. In a trio VR session organizing a cultural event, participants were represented by avatars, with or without the augmented model in a within-subject design. Results indicate significantly improved social presence and significant changes in participants’ behaviors, particularly their gaze, demonstrating the transformative potential of social augmentations for collaborative tasks in VR.

This paper investigates the impact of displaying the facial expressions of a user in real time on the performance of the task, the social presence and the behavioral changes of the users interacting in a Virtual Reality environment. To evaluate this approach, we conducted a study where the users collaborated to build a TV stand in dyad including a novice and an expert assistant. The collaborative task was divided into two independent phases: a face-to-face discussion phase without object manipulation, and a furniture assembly phase with object manipulation. Our results indicate that the proposed approach can increase social presence and lead to gaze behavior changes in multi-user environments during the face-to-face phase.

International audience

In a medical environment, coordination between medical staff is imperative. In cases in which a human doctor or medical coordinator is not present, patient care, particularly from non-experts, becomes more difficult. The difficulty increases when care is completed at a remote site, for example, on a manned mission to Mars. Communication capability from medical experts on Mars is limited. To address this problem, a medical assistant remote system is proposed to act as a coordinator between the humans present and the remote medical experts. A virtual agent assuming such a role will accept feedback from both, running the situation without errors and additional stress. Leadership styles will be employed by the agent to develop trust and perception of competence among its followers. Additionally, prediction of behaviour and situational changes by both medical professionals and by the agent are necessary in order to combat a 10-minute latency affecting communication between Earth and Mars.

This research work introduce virtual embodied tutors in Virtual Environments for learning devoted to learning of procedures for industrial systems. We present a communicative behavior which, integrated in pedagogical scenario, permits on the one hand to realize the pedagogical communicative actions at a semantic level (e.g., the tutor explains the goal of an action) and on the other hand to realize such actions through human-like communicative channels (i.e., the virtual tutor's voice, facial expressions and gestures). The communicative behavior relies on a taxonomy of questions in order to interpret the learner's communicative actions and to generate the tutor's own questions.

This paper presents a model that provides adaptive and evolutive interaction between a human and a virtual agent. After introducing the theoretical justifications, the aliveness metaphor and the notion of coupling are presented. Then, we propose a formalization of the model that relies on the temporal evolution of the coupling between participants and the existence of phases during the interaction. An example on a fitness exergame is provided and some illustrations show the behavior of the model during an interaction. A video complements this example.

Virtual Reality and immersive experiences, which allow players to share the same virtual environment as the characters of a virtual world, have gained more and more interest recently. In order to conceive these immersive virtual worlds, one of the challenges is to give to the characters that populate them the ability to express behaviors that can support the immersion. In this work, we propose a model capable of controlling and simulating a conversational group of social agents in an immersive environment. We describe this model which has been previously validated using a regular screen setting and we present a study for measuring whether users recognized the attitudes expressed by virtual agents through the realtime generated animations of nonverbal behavior in an immersive setting. Results mirrored those of the regular screen setting thus providing further insights for improving players experiences by integrating them into immersive simulated group conversations with characters that express different interpersonal attitudes.

This paper describes a substantial effort to build a real-time interactive multimodal dialogue system with a focus on emotional and non-verbal interaction capabilities. The work is motivated by the aim to provide technology with competences in perceiving and producing the emotional and non-verbal behaviours required to sustain a conversational dialogue. We present the Sensitive Artificial Listener (SAL) scenario as a setting which seems particularly suited for the study of emotional and non-verbal behaviour, since it requires only very limited verbal understanding on the part of the machine. This scenario allows us to concentrate on non-verbal capabilities without having to address at the same time the challenges of spoken language understanding, task modeling etc. We first summarise three prototype versions of the SAL scenario, in which the behaviour of the Sensitive Artificial Listener characters was determined by a human operator. These prototypes served the purpose of verifying the effectiveness of the SAL scenario and allowed us to collect data required for building system components for analysing and synthesising the respective behaviours. We then describe the fully autonomous integrated real-time system we created, which combines incremental analysis of user behaviour, dialogue management, and synthesis of speaker and listener behaviour of a SAL character displayed as a virtual agent. We discuss principles that should underlie the evaluation of SAL-type systems. Since the system is designed for modularity and reuse, and since it is publicly available, the SAL system has potential as a joint research tool in the affective computing research community.

Convincing conversational agents require a coherent set of behavioral responses that can be interpreted by a human observer as indicative of a personality. This paper discusses the continued development and subsequent evaluation of virtual agents based on sound psychological principles. We use Eysenck's theoretical basis to explain aspects of the characterization of our agents, and we describe an architecture where personality affects the agent's global behavior quality as well as their back-channel productions. Drawing on psychological research, we evaluate perception of our agents' personalities and credibility by human viewers (N = 187). Our results suggest that we succeeded in validating theoretically grounded indicators of personality in our virtual agents, and that it is feasible to place our characters on Eysenck's scales. A key finding is that the presence of behavioral characteristics reinforces the prescribed personality profiles that are already emerging from the still images. Our long-term goal is to enhance agents' ability to sustain realistic interaction with human users, and we discuss how this preliminary work may be further developed to include more systematic variation of Eysenck's personality scales.

This paper describes a substantial effort to build a real-time interactive multimodal dialogue system with a focus on emotional and nonverbal interaction capabilities. The work is motivated by the aim to provide technology with competences in perceiving and producing the emotional and nonverbal behaviors required to sustain a conversational dialogue. We present the Sensitive Artificial Listener (SAL) scenario as a setting which seems particularly suited for the study of emotional and nonverbal behavior since it requires only very limited verbal understanding on the part of the machine. This scenario allows us to concentrate on nonverbal capabilities without having to address at the same time the challenges of spoken language understanding, task modeling, etc. We first report on three prototype versions of the SAL scenario in which the behavior of the Sensitive Artificial Listener characters was determined by a human operator. These prototypes served the purpose of verifying the effectiveness of the SAL scenario and allowed us to collect data required for building system components for analyzing and synthesizing the respective behaviors. We then describe the fully autonomous integrated real-time system we created, which combines incremental analysis of user behavior, dialogue management, and synthesis of speaker and listener behavior of a SAL character displayed as a virtual agent. We discuss principles that should underlie the evaluation of SAL-type systems. Since the system is designed for modularity and reuse and since it is publicly available, the SAL system has potential as a joint research tool in the affective computing research community.

This demonstration aims to showcase the recently completed SEMAINE system. The SEMAINE system is a publicly available, fully autonomous Sensitive Artificial Listeners (SAL) system that consists of virtual dialog partners based on audiovisual analysis and synthesis (see http://semaine.opendfki.de/wiki). The system runs in real-time, and combines incremental analysis of user behavior, dialog management, and synthesis of speaker and listener behavior of a SAL character, displayed as a virtual agent. The SAL characters intend to engage the user in a conversation by paying attention to the user's emotions and nonverbal expressions. The characters have their own emotionally defined personality. During an interaction, the characters attempt to create an emotional workout for the user by drawing her/him towards their dominant emotion, through a combination of verbal and nonverbal expressions.

Abstract We aim at the realization of an Embodied Conversational Agent able to interact naturally and emotionally with user. In particular, the agent should behave expressively. Specifying for a given emotion, its corresponding facial expression will not produce the sensation of expressivity. To do so, one needs to specify parameters such as intensity, tension, movement property. Moreover, emotion affects also lip shapes during speech. Simply adding the facial expression of emotion to the lip shape does not produce lip readable movement. In this paper we present a model based on real data from a speaker on which was applied passive markers. The real data covers natural speech as well as emotional speech. We present an algorithm that determines the appropriate viseme and applies coarticulation and correlation rules to consider the vocalic and the consonantal contexts as well as muscular phenomena such as lip compression and lip stretching. Expressive qualifiers are then used to modulate the expressivity of lip movement. Our model of lip movement is applied on a 3D facial model compliant with MPEG‐4 standard. Copyright © 2004 John Wiley & Sons, Ltd.