Designing mobile, persuasive, and motivational interactions for sustainable behaviour change remains a challenge. Setting daily step goals on a mobile application can initially stimulate participation, but both the number of steps walked each day and the use of mobile applications decrease over time, even when individuals report high levels of motivation. To address this issue, we propose the integration of tailored gamification as a motivational mechanism, grounded in psychological theories. Self-Determination Theory is used to promote competence, while Regulatory Focus Theory is used to support individual differences and for tailoring motivational interactions. Gamification aims at reinforcing motivation and action-taking. In this article, we explain how we designed a motivational mobile application that incorporates these gamification mechanisms, and we describe how we evaluated its use during 28 days with 37 users. Results suggest a decrease in amotivation levels, and that promotion-focused individuals find gamified elements more motivational than prevention-focused ones.

The mobile health market is rapidly developing, but few apps follow evidence-based guidelines. Literature recommends personalized systems grounded in behavioral science, involving healthcare professionals in design to maximize effectiveness. To address this, we propose a metamodel to guide designers. This article discusses its application to low back pain self-management, focusing on four patient profiles: Unmotivated, Cautious, Depressed, and Confident. We evaluated the app over one month with 60 users. Of these, 32 users received a version of the application tailored to their profile, and 28 users received a version of the application without tailoring (no recommendations or motivational messages). We assessed user experience, engagement and psychological characteristics involved in the behavior change process. Results showed satisfactory user experience, impact of tailoring on user behavior and features to reduce fears and false beliefs and increase self-efficacy. Further efforts are needed to increase user engagement and observe an impact on long-term behavior.

There is a rising interest in animating realistic virtual agents for multiple purposes in different domains. Such a task requires systems capable of generating complex mental states on par with human emotional complexity. Considering the high representational capacity of Generative Adversarial Networks (GANs), it is only natural to consider them in such applications. In this work, we propose a conditional GAN model for generating sequences of facial expressions of categorical complex emotions. Trained on a scarce and highly imbalanced dataset, the proposed model is able to generate realistic variable-length sequences in a single inference step. These expressions of emotional states, of which there are 24 in total, follow the Facial Actions Coding System (FACS) formatting. In the absence of meaningful objective evaluation methods, we propose a deep-learning-based metric to assess the realism of generated Action Unit (AU) sequences: the Action Unit Fréchet Inception Distance (AUFID). Objective and subjective results validate the realism of our generated samples.

While multiple studies suggest that social skills are considered as essential in promoting employment, literature highlights a deficit in these skills among workers with Autism Spectrum Disorder (ASD) for whom difficulties in communication and social interaction are often reported. In this article, we explain how we used a User-Centered Design to identify and define in detail the work-related social skills that need to be trained. The aim was to develop social scenarios to train work-related social skills relevant for people with ASD using a digital app on a tablet. Participatory Design in our study included a needs analysis with questionnaires and interviews, as well as focus-groups and workshops with professionals of several disability employment services and workers or job seekers with ASD. Sixty-six social scenarios were written and 28 social skills were identified as relevant. This approach allowed us to design a tablet-based intervention which is an intelligent system by design since its content and structure target the social skills which require training for users with ASD.

The COPALZ model [3] is designed to simulate emotional disorders of a virtual agent representing a patient in a pedagogical scenario for training healthcare professionals. The identification of emotional and expressive pathologies may sometimes require an assessment over multiple interactions with trainees, as behaviors associated with emotional disorders are not systematically observed on patients' behavior in the early stages of the pathology. The aim of this article is to propose an evaluation method for this model, which, in the case of computational models of affects that generate nonverbal behaviors, requires a tailored approach. This task can be difficult as the correspondence between a pathology and observed behaviors is not systematic. Our method focuses on the pedagogical dimension and on the ability of the model to display pathological behaviors identified as relevant for training interactions. The results highlight the ability of the studied model to simulate multiple relevant pedagogical situations and adapt the virtual patient's behaviors to the evolution of the pathology and the patient's mood instability. This method gives interesting perspectives for the evaluation of virtual patients and computational models of affect.

Automatic analysis of job interviews has gained in interest amongst academic and industrial research. The particular case of asynchronous video interviews allows to collect vast corpora of videos where candidates answer standardized questions in monologue videos, enabling the use of deep learning algorithms. On the other hand, state-of-the-art approaches still face some obstacles, among which the fusion of information from multiple modalities and the interpretability of the predictions. We study the task of predicting candidates performance in asynchronous video interviews using three modalities (verbal content, prosody and facial expressions) independently or simultaneously, using data from real interviews which take place in real conditions. We propose a sequential and multimodal deep neural network model, called Multimodal HireNet. We compare this model to state-of-the-art approaches and show a clear improvement of the performance. Moreover, the architecture we propose is based on attention mechanism, which provides interpretability about which questions, moments and modalities contribute the most to the output of the network. While other deep learning systems use attention mechanisms to offer a visualization of moments with attention values, the proposed methodology enables an in-depth interpretation of the predictions by an overall analysis of the features of social signals contained in these moments.

Low back pain is a major public health problem that is the leading cause of disability in most countries. The recommendations are to propose personalized physical activity programs according to the individual characteristics (motivation, preferences, attitude towards physical activity) of each patient. However, the current management of low back pain includes rehabilitation programs that are costly and/or not always personalized. The objective of this research is to better understand the characteristics of people with chronic low back pain to identify typical profiles to inform the design of a tailored coaching mobile application. We selected eight concepts and associated scales to characterize the population: general health status (MYMOP), treatment burden (TBQ), quality of life (EQ-5D-5L), level (IPAQ), fears and beliefs (FABQ-AP) and motivation (EMAPS) for physical activity, emotional state (PHQ-9) and feeling of self-efficacy (FC-CPSES). These questionnaires were administered online via an e-cohort of low back pain patients. 193 patients with chronic low back pain were included. A clustering analysis allowed us to distinguish 4 profiles: (1) “Unmotivated,” characterized by the lowest motivation and the highest amotivation toward physical activity (i.e., they do not understand why it is useful), (2) “Cautious” with low motivation and having the highest fears and beliefs toward physical activity, (3) “Depressed” with good intrinsic motivation but also a significant level of depressive symptoms, and (4) “Confident” also showing good intrinsic motivation and having the highest self-efficacy. These four profiles of patients will be used to design a coaching mobile application tailored along the priorities (e.g., physical activity, pain management) and types of motivational messages displayed on the tailored mobile application.

Behavior change is one of the most important goals in psychotherapy. This study focuses on Motivational Interviewing (MI), which is collaborative communication aimed at eliciting the client’s own reasons for behavior change. To investigate the effectiveness of facial information in modeling MI, we collected an MI encounter corpus with speech and video data in the nutrition and fitness domains and annotated client utterances using the Manual for the Motivational Interviewing Skill Code (MISC). By analyzing client answers to the questions after the session, we found that clients who expressed more Change Talk were more motivated to change their behavior than those who expressed less Change Talk. We then proposed RNN-based multimodal models to detect Change Talk by setting a 2-class classification task: "Change Talk" and "not Change Talk." Our experiment showed that the best performing model was a multimodal BiLSTM model that fused language and client facial information. We also found that fusing language and facial information as context achieved better performance than the unimodal and no-context models. Moreover, we discuss the label imbalance problem and conduct an additional analysis using turns as a unit of analysis. As a result, our best model reached F1-score of 0.65 for Change Talk detection.

This workshop discusses how interactive, multimodal technology such as virtual agents can be used in social skills training for measuring and training social-affective interactions. Sensing technology now enables analyzing user’s behaviors and physiological signals. Various signal processing and machine learning methods can be used for such prediction tasks. Such social signal processing and tools can be applied to measure and reduce social stress in everyday situations, including public speaking at schools and workplaces.

Use of machine learning for automatic analysis of job interview videos has recently seen increased interest. Despite claims of fair output regarding sensitive information such as gender or ethnicity of the candidates, the current approaches rarely provide proof of unbiased decision-making, or that sensitive information is not used. Recently, adversarial methods have been proved to effectively remove sensitive information from the latent representation of neural networks. However, these methods rely on the use of explicitly labeled protected variables (e.g. gender), which cannot be collected in the context of recruiting in some countries (e.g. France). In this article, we propose a new adversarial approach to remove sensitive information from the latent representation of neural networks without the need to collect any sensitive variable. Using only a few frames of the interview, we train our model to not be able to find the face of the candidate related to the job interview in the inner layers of the model. This, in turn, allows us to remove relevant private information from these layers. Comparing our approach to a standard baseline on a public dataset with gender and ethnicity annotations, we show that it effectively removes sensitive information from the main network. Moreover, to the best of our knowledge, this is the first application of adversarial techniques for obtaining a multimodal fair representation in the context of video job interviews. In summary, our contributions aim at improving fairness of the upcoming automatic systems processing videos of job interviews for equality in job selection.

This workshop discusses how interactive, multimodal technology such as virtual agents can be used in social skills training for measuring and training social-affective interactions. Sensing technology now enables analyzing user’s behaviors and physiological signals. Various signal processing and machine learning methods can be used for such prediction tasks. Such social signal processing and tools can be applied to measure and reduce social stress in everyday situations, including public speaking at schools and workplaces.

New technologies for autism focus on the training of either social skills or motor skills, but not both. Such a dichotomy omits a wide range of joint action tasks that require the coordination of two persons (e.g. moving a heavy furniture). The training of these physical tasks performed in dyad has great potential to foster inclusiveness while having an impact on both social and motor skills. In this paper, we present the design of a tangible and virtual interactive system for the training of children with Autism Spectrum Disorder (ASD) in performing joint actions. The proposed system is composed of a virtual character projected onto a surface on which a tangible object is magnetized: both the user and the virtual character hold the object, thus simulating a joint action. We report and discuss preliminary results of a field training study, which shows the potential of the interactive system.

Abstract Postural interaction is of major importance during job interviews. While several prototypes enable users to rehearse for public speaking tasks and job interviews, few of these prototypes support subtle bodily interactions between the user and a virtual agent playing the role of an interviewer. The design of our system is informed by a multimodal corpus that was previously collected. In this paper, we explain how we were inspired by these video recordings of human interviewers to build a library of motion‐captured movements that interviewers are most likely to display. We designed a fully automatic interactive virtual agent able to display these movements in response to the bodily movements of the user. Thirty‐two participants presented themselves to this virtual agent during a simulated job interview. We focused on the self‐presentation task of the job interview, the virtual agent being listening. Participants stood on a force platform that recorded the displacements of their center of pressure to assess the postural impact of our design. We also collected video recordings of their movements and computed the contraction index and the quantity of motion of their bodies. We explain the different hypotheses that we made concerning (1) the comparison between the performance of participants with human interviewers and the performance of participants with virtual interviewers, (2) the comparison between mirror and random postural behaviors displayed by a female versus a male virtual interviewer, and (3) the correlation between the participants' performance and their personality traits. Our results suggest that users perceive the simulated self‐presentation task with the virtual interviewer as threatening and as difficult as the presentation task with the human interviewers. Furthermore, when users interact with a virtual interviewer that mirrors their postures, these users perceive the interviewer as being affiliative. Finally, a correlation analysis showed that personality traits had a significant relation to the postural behaviors and performance of the users during their presentation.

This workshop discusses how interactive, multimodal technology such as virtual agents can be used in social skills training for measuring and training social-affective interactions. Sensing technology now enables analyzing user's behaviors and physiological signals. Various signal processing and machine learning methods can be used for such prediction tasks. Such social signal processing and tools can be applied to measure and reduce social stress in everyday situations, including public speaking at schools and workplaces.

Previous studies about kinesthetic expressions of emotions are mainly based on acted expressions of emotions, which can be quite different from spontaneous expressions. This paper describes a study involving a stress induction procedure and stress perception through haptic expressions with Ntotal= 41 young men (aged 19 - 37), all right-handed. We designed a game application to collect spontaneous expressions of stress. This application involved haptic interactions and a stressful event. We observed changes in the haptic behaviors of the participants over different phases: before, during and after the stress induction. In the next step, we investigated how the collected haptic behaviors (both kinematic and force components) were haptically perceived by the other participants. The results suggest the ability of kinesthetic expressions to communicate a spontaneous stress from one person to another.

New technologies drastically change recruitment techniques. Some research projects aim at designing interactive systems that help candidates practice job interviews. Other studies aim at the automatic detection of social signals (e.g. smile, turn of speech, etc...) in videos of job interviews. These studies are limited with respect to the number of interviews they process, but also by the fact that they only analyze simulated job interviews (e.g. students pretending to apply for a fake position). Asynchronous video interviewing tools have become mature products on the human resources market, and thus, a popular step in the recruitment process. As part of a project to help recruiters, we collected a corpus of more than 7000 candidates having asynchronous video job interviews for real positions and recording videos of themselves answering a set of questions. We propose a new hierarchical attention model called HireNet that aims at predicting the hirability of the candidates as evaluated by recruiters. In HireNet, an interview is considered as a sequence of questions and answers containing salient socials signals. Two contextual sources of information are modeled in HireNet: the words contained in the question and in the job position. Our model achieves better F1-scores than previous approaches for each modality (verbal content, audio and video). Results from early and late multimodal fusion suggest that more sophisticated fusion schemes are needed to improve on the monomodal results. Finally, some examples of moments captured by the attention mechanisms suggest our model could potentially be used to help finding key moments in an asynchronous job interview.

The impact of non verbal behaviour in a hiring decision remains an open question. Investigating this question is important, as it could provide a better understanding on how to train candidates for job interviews and make recruiters be aware of influential non verbal behaviour. This research has recently been accelerated due to the development of tools for the automatic analysis of social signals (facial expression detection, speech processing, etc), and the emergence of machine learning methods. However, these studies are still mainly based on hand engineered features, which imposes a limit to the discovery of influential social signals. On the other side, deep learning methods are a promising tool to discover complex patterns without the necessity of feature engineering. In this paper, we focus on studying influential non verbal social signals in asynchronous job video interviews that are discovered by deep learning methods. We use a previously published deep learning system that aims at inferring the hirability of a candidate with regard to a sequence of interview questions. One particularity of this system is the use of attention mechanisms, which aim at identifying the relevant parts of an answer. Thus, information at a fine-grained temporal level could be extracted using global (at the interview level) annotations on hirability. While most of the deep learning systems use attention mechanisms to offer a quick visualization of slices when a rise of attention occurs, we perform an in-depth analysis to understand what happens during these moments. First, we propose a methodology to automatically extract slices where there is a rise of attention (attention slices). Second, we study the content of attention slices by comparing them with randomly sampled slices. Finally, we show that they bear significantly more information for hirability than randomly sampled slices, and that such information is related to visual cues associated with anxiety and turn taking.

Managing a medical team in emergency situations requires not only technical but also non-technical skills. Leaders must train to manage different types of subordinates, and how these subordinates will respond to orders and stressful events. Before designing virtual training environments for these leaders, it is necessary to understand how leaders perceive the nonverbal behaviors of virtual characters playing the role of subordinates. In this article, we describe a study we conducted to explore how leaders categorize virtual subordinates from the non-verbal expressions they display (i.e., facial expressions, torso orientation, gaze direction). We analyze how these multimodal behaviors impact the perception of follower style (proactive vs. passive, insubordination), interpersonal attitudes (dominance vs. submission) and stress. Our results suggest that leaders categorize virtual subordinates via nonverbal behaviors that are also perceived as signs of stress and interpersonal attitudes.

Health care delivery in military conflict, in peacekeeping missions or in the aftermath of disaster, implies high stress environments with danger exposures, life-threatening events and high levels of work demand. Crisis and emergency risk communication remains a real challenge. The rapid response of emergency medical teams has an important role to play in preventing serious adverse events. During critical events, medical errors can be related to human or system factors, including ineffective team leadership, non-standardized team communication, a lack of global situation awareness, poor use of resources and inappropriate triage and prioritization. The VICTEAMS project aims at building a virtual environment for training rescue team leaders to non-technical skills. Depending on the leader abilities, the virtual characters playing the subordinates (e.g. nurses) reproduce a variety of behaviors like erroneous actions, followership attitudes or stress-based behaviors. A pedagogical director tailors the difficulty according to the dynamic profile of the learner.

In this article, effectiveness of air jet stimulation in mediated emotional communication was investigated by assessing cross-modal influences of visual emotional expressions on tactile perception. Brain responses to combined visual faces and air jet stimuli were measured using event-related potentials; whereas, emotional responses were assessed using self-reported pleasantness of the tactile stimulation. ERP results reveal significant differences between the different facial expressions for the same tactile air-jet intensity in the somatosensory area. Moreover, participants’ pleasantness ratings suggest an effect of the visual stimulus on the difference tactile conditions that correspond to air jet stimulation intensities: low, medium, and high. These promising results provide evidence in the potential efficiency of this stimulation technique in activating skin receptors that play an important role in social and affective behaviors.

We conducted an experimental study to investigate how the participants combine touch and facial expression to evaluate emotional valence. In this study, visual and haptic stimuli were presented separately and then presented together. The visual stimuli comprised pictures of facial expressions with different emotional levels, and the touch stimuli consisted of air jet tactile stimulation performed on the arms of the participants. The participants were asked to evaluate the communicated emotional valence on a continuous scale. Information Integration Theory was used to model the algebraic rule (addition, multiplication, averaging, and so on) that underlies the multimodal perception of emotional valence. Analyses showed that the participants usually integrated both visual and touch information to evaluate the emotional valence. The main integration rule was averaging, with nonsystematic predominance of each modality over the other modality.

The aim of the present study is to identify relevant nonverbal features allowing the discrimination of different stressful behaviors, with the consideration of personality factors. In order to achieve this aim, we propose a new method for psychological stress induction involving four different stressful tasks. The proposed protocol was tested with 45 PhD students and the analysis of heart rate variability suggests that stress was indeed elicited. PhD students were selected as participants because they often experience stress. Multimodal data was collected and analyzed in order to identify nonverbal behavioral features related to the different stressful tasks. The psychological profile of participants was taken into account to understand how different stressful behaviors are correlated with personality factors. Results suggest that relevant nonverbal behaviors can discriminate between stressful tasks. In addition, relevant behaviors involving movement variability appear to be correlated with personality factors and stressful tasks.

Imitation is of major importance during social interactions, would it be between humans or between a human and a robot. This is even more true when considering users with special needs. In this paper, we describe an experimental imitation task protocol using a robot Nao that we designed to assess whether sensory profiles of children with Autistic Spectrum Disorder (ASD) influence their capabilities to imitate or to initiate gestures that are going to be imitated. We based our work on the hypothesis that children with an overreliance on proprioceptive cues and hyporeactivity to visual cues have a greater difficulty imitating and improve their skills more slowly than children with an overreliance on visual cues and hyporeactivity to proprioceptive cues. Twelve children and teenagers with ASD participated in seven imitation sessions over eight weeks. As expected, we observed that children with an overreliance on proprioceptive cues and hyporeactivity to visual cues had more difficulties imitating the robot than the other children. Moreover, the repeated sessions revealed to have positive effects on social behaviors displayed by all children (gaze to the partner, imitations) toward a human partner after the sessions with the robot. We conclude on the possible impacts of such results on the design of social human-robot interactions for users with ASD.

Individuals suffering from Autistic Spectrum Disorder (ASD) have impaired skills in social communication and joint attention. In this paper, we explain how we designed and evaluated a Joint Attention (JA) task for individuals with ASD using the Nao humanoid robot. The interaction was tested in children and teenagers with ASD (N=11). Their proprioceptive and visual integration of cues were first assessed, with the hypothesis that individuals with an overreliance on proprioceptive cues and with a hyporeactivity to visual cues would have more difficulties conducting successful interactions with the robot. We observed that participants with an overreliance on proprioceptive cues and hyporeactivity to visual cues showed different behaviors in responding to joint attention. They followed the prompting of the Nao robot more slowly than individuals with an overreliance on visual cues and a hyporeactivity to proprioceptive cues. Defining such individual profiles prior to the social interaction with a robot and working closely with caregivers could provide promising strategies for designing successful and adapted Human-Robot Interaction (HRI) for individuals with ASD.

Artificial agents are becoming artificial companions, interacting with the user on a long-term basis. This evolution brought new challenges to the affective computing domain, such as designing artificial agents with personalities to the benefits of the user. Endowing artificial agents with personality could help to increase the agent's believability, hence easing the interaction. This paper touches on two questions pertaining to computational personality modeling: 1/ how to produce artificial personalities which can inform personality researchers, whether from computer sciences or psychology and 2/ will behaviors produced by artificial agents be perceived by users as putting the programmed personality across as such. We propose to use a data-driven approach to endow artificial agents with personality, using the regulatory focus theory as a framework. We used machine-learned game strategies, in the form of alternative decision trees computed from human data, to convey the personality of artificial agents. We then tested whether these personalities can be perceived by users after playing a game against these agents. We used two artificial agents as controls: one randomly playing and one with an “average / depersonalized” strategy. On the one hand, our results show that agents' regulatory focus, when programmed, can be accurately perceived by users. On the other hand, our results also point out that personality will be perceived by users even if the agent's design does not intend to transmit one.

Previous studies about kinesthetic expressions of emotions are mainly based on acted expressions of affective states, which might be quite different from spontaneous expressions. In a previous study, we proposed a task to collect haptic expressions of a spontaneous stress. In this paper, we explore the effectiveness of this task to induce a spontaneous stress in two ways: a subjective feedback, and a more objective approach-avoidance behavior.

The use of virtual avatars, through facial or gestural expressions, is considered to be a main support for affective communication. Currently, different works have studied the potential of a kinesthetic channel for conveying such information. However, they still have not investigated the complementarity between visual and kinesthetic feedback to effectively convey emotion. This paper studies the relation between some emotional dimensions and the visual and kinesthetic modalities. The experimental results show that subjects used visual and kinesthetic feedbacks to evaluate the pleasure and the arousal dimensions, respectively. We also observed a link between the recognition rate of emotions expressed with the visual modality (resp. kinesthetic modality) and the magnitude of that emotion's pleasure dimension (resp. arousal dimension). These different results should help in the selection of feedbacks according to the features of the investigated emotion.

This study presents an experiment highlighting how participants combine facial expressions and haptic feedback to perceive emotions when interacting with an expressive humanoid robot. Participants were asked to interact with the humanoid robot through a handshake behavior while looking at its facial expressions. Experimental data were examined within the information integration theory framework. Results revealed that participants combined Facial and Haptic cues additively to evaluate the Valence, Arousal, and Dominance dimensions. The relative importance of each modality was different across the emotional dimensions. Participants gave more importance to facial expressions when evaluating Valence. They gave more importance to haptic feedback when evaluating Arousal and Dominance.

This paper deals with the haptic affective social interaction during a greeting handshaking between a human and a humanoid robot. The goal of this work is to study how the haptic interaction conveys emotions, and more precisely, how it in'uences the perception of the dimensions of emotions expressed through the facial expressions of the robot. Moreover, we examine the bene'ts of the multimodality (i.e., visuo-haptic) over the monomodality (i.e., visual-only and haptic-only). The experimental results with Meka robot show that the multimodal condition presenting high values for grasping force and joint stiffness are evaluated with higher values for the arousal and dominance dimensions than during the visual condition. Furthermore, the results corresponding to the monomodal haptic condition showed that participants discriminate well the dominance and the arousal dimensions of the haptic behaviours presenting low and high values for grasping force and joint stiffness.

In this paper, we describe the elaboration and the validation of a body and face database1, of 96 videos of 1 to 2 seconds of duration, expressing 4 emotions (i.e., anger, happiness, fear, and sadness) elicited through 4 platforms of increased visual complexity and level of embodiment. The final aim of this database is to develop an individualized training program designed for individuals suffering of autism in order to help them recognize various emotions on different test platforms: two robots, a virtual agent, and a human. Before assessing the recognition capabilities of individuals with ASD, we validated our video database on typically developed individuals (TD). Moreover, we also looked at the relationship between the recognition rate and their personality traits (extroverted (EX) vs. introverted (IN)). We found that the personality of our TD participants did not lead to a different recognition behavior. However, introverted individuals better recognized emotions from less visually complex characters than extroverted individuals.

Virtual sport coaches guide users through their physical activity and provide motivational support. Users’ motivation can rapidly decay if the movements of the virtual coach are too stereotyped. Kinematic patterns generated while performing a predefined fitness movement can elicit and help to prolong users’ interaction and interest in training. Human body kinematics has been shown to convey various social attributes such as gender, identity, and acted emotions. To date, no study provides information regarding how spontaneous emotions and personality traits together are perceived from full-body movements. In this article, we study how people make reliable inferences regarding spontaneous emotional dimensions and personality traits of human coaches from kinematic patterns they produced when performing a fitness sequence. Movements were presented to participants via a virtual mannequin to isolate the influence of kinematics on perception. Kinematic patterns of biological movement were analyzed in terms of movement qualities according to the effort-shape [Dell 1977] notation proposed by Laban [1950]. Three studies were performed to provide an analysis of the process leading to perception: from coaches’ states and traits through bodily movements to observers’ social perception. Thirty-two participants (i.e., observers) were asked to rate the movements of the virtual mannequin in terms of conveyed emotion dimensions, personality traits (five-factor model of personality), and perceived movement qualities (effort-shape) from 56 fitness movement sequences. The results showed high reliability for most of the evaluated dimensions, confirming interobserver agreement from kinematics at zero acquaintance. A large expressive halo merging emotional (e.g., perceived intensity) and personality aspects (e.g., extraversion) was found, driven by perceived kinematic impulsivity and energy. Observers’ perceptions were partially accurate for emotion dimensions and were not accurate for personality traits. Together, these results contribute to both the understanding of dimensions of social perception through movement and the design of expressive virtual sport coaches.

Haptic expression of emotions has received less attention than other modalities. Bonnet et al. [2011] combine visio-haptic modalities to improve the recognition and discrimination of some emotions. However, few works investigated how these modalities complement each other. For instance, Bickmore et al. [2010] highlight some non-significant tendencies of complementarity between the visual and haptic modalities.

In this article the role of different categories of postures in the detection, recognition, and interpretation of emotion in contextually rich scenarios, including ironic items, is investigated. Animated scenarios are designed with 3D virtual agents in order to test 3 conditions: In the “still” condition, the narrative content was accompanied by emotional facial expressions without any body movements; in the “idle” condition, emotionally neutral body movements were introduced; and in the “congruent” condition, emotional body postures congruent with the character's facial expressions were displayed. Those conditions were examined by 27 subjects, and their impact on the viewers’ attentional and emotional processes was assessed. The results highlight the importance of the contextual information to emotion recognition and irony interpretation. It is also shown that both idle and emotional postures improve the detection of emotional expressions. Moreover, emotional postures increase the perceived intensity of emotions and the realism of the animations.

Several studies have investigated the relevance of haptics to convey various types of emotions physically. This article investigates the improvement of the recognition rate of emotions using visuo-haptic feedback compared to facial and haptic expressions alone. Four experiments were conducted in which the recognition rates of emotions using facial, haptic and visuo-haptic expressions were tested. The first experiment evaluates the recognition rate of emotions using facial expressions. The second experiment collects a large corpus of 3D haptic expressions of certain emotions and subsequently identifies the relevant haptic expression for each emotion. The third experiment evaluates the selected haptic expressions through statistical and perceptive tests to retain the ones that result in the most accurate identification of the corresponding emotion. Finally, the fourth experiment studies the effect of visuo–haptic coupling on the recognition of the investigated emotions. Generally, emotions with high amplitudes of pleasure are better recognized in the visual modality. However, emotions with high activation are better recognized in the haptic modality. These results also highlighted the finding that participants are not equally aided by each modality when recognizing emotions efficiently. Beyond the recognition rate, multimodal expressions improved the sensation of presence and expressivity.

This article analyses the issues pertaining to the simulation of joint attention with virtual humans. Gaze represents a powerful communication channel illustrated by the pivotal role of joint attention in social interactions. To our knowledge, there have been only few attempts to simulate gazing patterns associated with joint attention as a mean for developing empathic virtual agents. Eye-tracking technologies now enable creating non-invasive gaze-contingent systems that empower the user with the ability to lead a virtual human's focus of attention in real-time. Although gaze control can be deliberate, most of our visual behaviors in everyday life are not. This article reports empirical data suggesting that users only have partial awareness of controlling gaze-contingent displays. The technical challenges induced by detecting the user's focus of attention in virtual reality are reviewed and several solutions are compared. We designed and tested a platform for creating virtual humans endowed with the ability to follow the user's attention. The article discusses the advantages of simulating joint attention for improving interpersonal skills and user engagement. Joint attention plays a major role in the development of autism. The platform we designed is intended for research and treatment of autism and tests included participants with this disorder.

This workshop covers real-time computational techniques for the recognition and interpretation of human affective and social behaviour, and techniques for synthesis of believable social behaviour supporting real-time adaptive human-agent and human-robot interaction in real-world environments.

Consistency and coherence in behaviors are key concepts for creating believable virtual agents. Personality is described by psychology as a set of stable and individualized behaviors. Thus, endowing agents with a personality can help agents to exhibit consistent behaviors. In this paper, we suggest using a socio-cognitive perspective to build a model of personality for virtual agents and to highlight some of the personality structures and processes. Because many researchers in personality psychology emphasize the role of the self in personality, we propose PERSEED, a conceptual model of personality that is based on self-regulation theories.

Eliciting emotions to collect affective expressions is frequently used in studies about facial, spoken or bodily expressions of emotions. However, recent works in affective hap tics focus mostly on expressions of acted emotions. The expression of spontaneous emotions using the haptic modality remains unexplored. This paper discusses methodological considerations for designing experimental protocols to collect haptic expressions of spontaneous emotions. We highlight the main difficulties to solve, provide illustrative examples from several experiments that we conducted, and describe methodological considerations for eliciting emotions and collecting corresponding haptic expressions.

Postural control is a dynamical process that has been extensively studied in motor control research. Recent experimental work shows a direct impact of affects on human balance. However, few studies on the automatic recognition of affects in full body expressions consider balance variables such as center of gravity displacements. Force plates enable the capture of balance variables with high precision. Automatic video extraction of the center of gravity is a basic alternative, which can be easily accessible for a wide range of public applications. This paper presents a comparison of balance variables extracted from the force plate and video processing. These variables are used to capture the bodily expressions of participants in a public speaking task designed to elicit stress. Results show that the variability of the center of gravity displacements from the force plate and video are related to negative emotions and situation appraisals. The power spectrum density broadness of the center of pressure from the force plate is related to Difficulty Describing Feelings, an important factor from a dispositional trait of Alexithymia. Implications of the use of such methods are discussed.

Databases of spontaneous multimodal expressions of affective states occurring during a task are few. This paper presents a protocol for eliciting stress in a public speaking task. Behaviors of 19 participants were recorded via a multimodal setup including speech, video of the facial expressions and body movements, balance via a force plate, and physiological measures. Questionnaires were used to assert emotional states, personality profiles and relevant coping behaviors to study how participants cope with stressful situations. Several subjective and objective performances were also evaluated. Results show a significant impact of the overall task and conditions on the participants' emotional activation. The possible future use of this new multimodal emotional corpus is described.

Several studies in affective computing consider the role of haptic channel via the tactile and kinesthetic components. However, these works used constraining haptic devices with limited stimulation strategies. This paper proposes to investigate a new haptic device for the tactile stimulation with a mobile air jet. This approach is more suitable for the stimulation of skin mechanoreceptors which play an important role in social and affective behaviors. An experimental study was conducted in order to understand the relationships between the physical features of the air jet and the perception of emotions by participants. The results highlight a main effect of the intensity and movement velocity of the air jet stimulation on the evaluations of valence, arousal, and dominance.

MACH--My Automated Conversation coacH--is a novel system that provides ubiquitous access to social skills training. The system includes a virtual agent that reads facial expressions, speech, and prosody and responds with verbal and nonverbal behaviors in real time. This paper presents an application of MACH in the context of training for job interviews. During the training, MACH asks interview questions, automatically mimics certain behavior issued by the user, and exhibit appropriate nonverbal behaviors. Following the interaction, MACH provides visual feedback on the user's performance. The development of this application draws on data from 28 interview sessions, involving employment-seeking students and career counselors. The effectiveness of MACH was assessed through a weeklong trial with 90 MIT undergraduates. Students who interacted with MACH were rated by human experts to have improved in overall interview performance, while the ratings of students in control groups did not improve. Post-experiment interviews indicate that participants found the interview experience informative about their behaviors and expressed interest in using MACH in the future.

Analysis of non-verbal behaviors in HCI allows understanding how individuals apprehend and adapt to different situations of interaction. This seems particularly relevant when considering tasks such as speaking in a foreign language, which is known to elicit anxiety. This is even truer for young users for whom negative pedagogical feedbacks might have a strong negative impact on their motivation to learn.

Several studies have investigated the relevance of haptics to physically convey various types of emotion. However, they use basic analysis approaches to identify the relevant features for an effective communication of emotion. This article presents an advanced analysis approach, based on the clustering technique, that enables the extraction of the general features of affective haptic expressions as well as the identification of specific features in order to discriminate between close emotions that are difficult to differentiate. This approach was tested in the context of affective communication through a virtual handshake. It uses a haptic device, which enables the expression of 3D movements. The results of this research were compared to those of the standard Analysis of Variance method in order to highlight the advantages and limitations of each approach.

Several studies showed the relevance of haptics for conveying emotions to users. These studies usually cover recognition rate of emotions from haptic expressions. Surprisingly, the analysis of features of these haptic expressions is in counterpart often limited to a classical analysis of variance. However, this method suffers of several limitations for the analysis of such data. This paper investigates the complementarity of three different statistical methods for the processing of haptic expressions of emotions.

Exposure to distressing computer-generated stimuli and feedback of physiological changes during exposure have been effective in the treatment of anxiety disorders (e.g., social phobia). Here we studied voluntary facial activations as a method for regulating more spontaneous physiological changes during virtual social stimulation. Twenty-four participants with a low or high level of social anxiety activated either the corrugator supercilii (used in frowning) or the zygomaticus major (used in smiling) facial muscle to keep a female or a male computer character walking towards them. The more socially anxious participants had a higher level of skin conductance throughout the trials as compared to less anxious participants. Within both groups, short-term skin conductance responses were enhanced both during and after facial activations; and corrugator supercilii activations facilitated longer term electrodermal relaxation. Zygomaticus major activations had opposite effects on subjective emotional ratings of the less and the more socially anxious. In sum, voluntary facial activations were effective in regulating emotional arousal during virtual social exposure. Corrugator supercilii activation was found an especially promising method for facilitating autonomic relaxation.

Affect-sensitive systems such as social robots and virtual agents are increasingly being investigated in real-world settings. In order to work effectively in natural environments, these systems require the ability to infer the affective and mental states of humans and to provide appropriate timely output that helps to sustain long-term interactions. This special issue, which appears in two parts, includes articles on the design of socio-emotional behaviors and expressions in robots and virtual agents and on computational approaches for the automatic recognition of social signals and affective states.

Location-aware ambient environments are relevant in designing interactive virtual characters. However, they raise several questions about spatial behaviors to be displayed by virtual characters, and about the perception of virtual characters by users. We designed a location-aware virtual agent that adapts its spatial behavior to users' and objects' locations during a search task in a smart room. We conducted an experimental evaluation comparing this adaptive agent with an agent that does not perceive nor use the location of users and objects. The location-aware adaptive agent elicited higher levels of perceived presence and perceived adaptivity. Furthermore, performance was less influenced by task difficulty when users interacted with the adaptive agent. Such results can be useful for the design of future location-aware virtual agents.

The 3rd International Workshop on Affective Interaction in Natural Environments, AFFINE, follows a number of successful AFFINE workshops and events commencing in 2008.A key aim of AFFINE is the identification and investigation of significant open issues in real-time, affect-aware applications 'in the wild' and especially in embodied interaction, for example, with robots or virtual agents. AFFINE seeks to bring together researchers working on the real-time interpretation of user behaviour with those who are concerned with social robot and virtual agent interaction frameworks.

This work presents the development of a real-time framework for the research of multimodal feedback of robots/talking agents in the context of Human Robot Interaction (HRI) and Human Computer Interaction (HCI). For evaluating the framework, a Multimodal corpus is built (ENTERFACE_STEAD), and a study on the important multimodal features was done for building an active Robot/Agent listener of a storytelling experience with Humans. The experiments show that even when building the same reactive behavior models for Robot and Talking Agents, the interpretation and the realization of the behavior communicated is different due to the different communicative channels Robots/Agents offer be it physical but less human-like in Robots, and virtual but more expressive and human-like in Talking agents.

Until now, research on arrangement of verbal and non-verbal information in multimedia presentations has not considered multimodal behavior of animated agents. In this paper, we will present an experiment exploring the effects of different types of speech–gesture cooperation in agents’ behavior: redundancy (gestures duplicate pieces of information conveyed by speech), complementarity (distribution of information across speech and gestures) and a control condition in which gesture does not convey semantic information. Using a Latin-square design, these strategies were attributed to agents of different appearances to present different objects. Fifty-four male and 54 female users attended three short presentations performed by the agents, recalled the content of presentations and evaluated both the presentations and the agents. Although speech–gesture cooperation was not consciously perceived, it proved to influence users’ recall performance and subjective evaluations: redundancy increased verbal information recall, ratings of the quality of explanation, and expressiveness of agents. Redundancy also resulted in higher likeability scores for the agents and a more positive perception of their personality. Users’ gender had no influence on this set of results.

In everyday life, speech is part of a multichannel system involved in conveying emotion. Understanding how it operates in that context requires suitable data, consisting of multimodal records of emotion drawn from everyday life. This paper reflects the experience of two teams active in collecting and labelling data of this type. It sets out the core reasons for pursuing a multimodal approach, reviews issues and problems for developing relevant databases, and indicates how we can move forward both in terms of data collection and approaches to labelling.

Recent advances in the specification of the multimodal behavior of Embodied Conversational Agents (ECA) have proposed a direct and deterministic one-step mapping from high-level specifications of dialog state or agent emotion onto low-level specifications of the multimodal behavior to be displayed by the agent (e.g. facial expression, gestures, vocal utterance). The difference of abstraction between these two levels of specification makes difficult the definition of such a complex mapping. In this paper we propose an intermediate level of specification based on combinations between modalities (e.g. redundancy, complementarity). We explain how such intermediate level specifications can be described using XML in the case of deictic expressions. We define algorithms for parsing such descriptions and generating the corresponding multimodal behavior of 2D cartoon-like conversational agents. Some random selection has been introduced in these algorithms in order to induce some "natural variations" in the agent's behavior. We conclude on the usefulness of this approach for the design of ECA.

Although there are different definitions of autonomous agents, one of their features should probably be the flexibility of choice an autonomous agent should have on how and with which other agents it should co-operate. The TYCOON framework is introduced for studying co-operation between agents. The framework includes on one hand a typology including six types of co-operation between agents (transfer, equivalence, specialization, redundancy, 'complementarity' and concurrency), and on the other hand the associate algorithms for computing metrics of co-operative behaviour. It is described how the framework can be applied to two different problem-solving tasks: a simple variable binding task and a more complex multimodal collective presentation task. I believe that these TYCOON low-level specifications of co-operation between agents might be useful as building blocks for bridging the gap between experimental human behaviour analysis and the modelling of higher-level co-operation-related attitudes related to autonomy, delegation and control.