There is a rising interest in animating realistic virtual agents for multiple purposes in different domains. Such a task requires systems capable of generating complex mental states on par with human emotional complexity. Considering the high representational capacity of Generative Adversarial Networks (GANs), it is only natural to consider them in such applications. In this work, we propose a conditional GAN model for generating sequences of facial expressions of categorical complex emotions. Trained on a scarce and highly imbalanced dataset, the proposed model is able to generate realistic variable-length sequences in a single inference step. These expressions of emotional states, of which there are 24 in total, follow the Facial Actions Coding System (FACS) formatting. In the absence of meaningful objective evaluation methods, we propose a deep-learning-based metric to assess the realism of generated Action Unit (AU) sequences: the Action Unit Fréchet Inception Distance (AUFID). Objective and subjective results validate the realism of our generated samples.

Creativity Support Tools (CSTs), like sketching tools, enable designers in their creative activities. As designers may have different needs during different thinking phases, accessing physiological responses in real-time would enable CSTs to better adapt to these different phases. Several studies have focused on either different phases on the same tool or the same phase on different tools. We investigate ways to discriminate divergent and convergent thinking phases using physiological response. We replicated the same experiment with thirty participants each on two different CSTs: in immersive virtual reality and on an interactive whiteboard (2D). We analyzed Heart Rate Variability (HRV) and Galvanic Skin Response (GSR) during a divergent thinking task followed by a convergent thinking task. Our results revealed significant changes in arousal, measured through HRV and GSR, which seem specific to each creative thinking phase. We discuss how each phase can be linked with specific reactions independently of the CST. We suggest that it may be possible to detect a favorable creative state depending on the task being performed.

Abstract Postural interaction is of major importance during job interviews. While several prototypes enable users to rehearse for public speaking tasks and job interviews, few of these prototypes support subtle bodily interactions between the user and a virtual agent playing the role of an interviewer. The design of our system is informed by a multimodal corpus that was previously collected. In this paper, we explain how we were inspired by these video recordings of human interviewers to build a library of motion‐captured movements that interviewers are most likely to display. We designed a fully automatic interactive virtual agent able to display these movements in response to the bodily movements of the user. Thirty‐two participants presented themselves to this virtual agent during a simulated job interview. We focused on the self‐presentation task of the job interview, the virtual agent being listening. Participants stood on a force platform that recorded the displacements of their center of pressure to assess the postural impact of our design. We also collected video recordings of their movements and computed the contraction index and the quantity of motion of their bodies. We explain the different hypotheses that we made concerning (1) the comparison between the performance of participants with human interviewers and the performance of participants with virtual interviewers, (2) the comparison between mirror and random postural behaviors displayed by a female versus a male virtual interviewer, and (3) the correlation between the participants' performance and their personality traits. Our results suggest that users perceive the simulated self‐presentation task with the virtual interviewer as threatening and as difficult as the presentation task with the human interviewers. Furthermore, when users interact with a virtual interviewer that mirrors their postures, these users perceive the interviewer as being affiliative. Finally, a correlation analysis showed that personality traits had a significant relation to the postural behaviors and performance of the users during their presentation.

Self-tracking aims to increase awareness, decrease undesired behaviors, and ultimately lead towards a healthier lifestyle. However, inappropriate communication of self- tracking results might cause the opposite effect. Subtle self- tracking feedback is an alternative that can be provided with the aid of an artificial agent representing the self. Hence, we propose a wearable pet that reflects the user's affective states through visual and haptic feedback. By eliciting empathy and fostering helping behaviors towards it, users would indirectly help themselves. A wearable prototype was built, and three user studies performed to evaluate the appropriateness of the proposed affective representations. Visual representations using facial and body cues were clear for valence and less clear for arousal. Haptic interoceptive patterns emulating heart-rate levels matched the desired feedback urgency levels with a saturation frequency. The integrated visuo-haptic representations matched to participants own affective experience. From the results, we derived three design guidelines for future robot mirroring wearable systems: physical embodiment, interoceptive feedback, and customization.

The aim of the present study is to identify relevant nonverbal features allowing the discrimination of different stressful behaviors, with the consideration of personality factors. In order to achieve this aim, we propose a new method for psychological stress induction involving four different stressful tasks. The proposed protocol was tested with 45 PhD students and the analysis of heart rate variability suggests that stress was indeed elicited. PhD students were selected as participants because they often experience stress. Multimodal data was collected and analyzed in order to identify nonverbal behavioral features related to the different stressful tasks. The psychological profile of participants was taken into account to understand how different stressful behaviors are correlated with personality factors. Results suggest that relevant nonverbal behaviors can discriminate between stressful tasks. In addition, relevant behaviors involving movement variability appear to be correlated with personality factors and stressful tasks.

Previous studies about kinesthetic expressions of emotions are mainly based on acted expressions of affective states, which might be quite different from spontaneous expressions. In a previous study, we proposed a task to collect haptic expressions of a spontaneous stress. In this paper, we explore the effectiveness of this task to induce a spontaneous stress in two ways: a subjective feedback, and a more objective approach-avoidance behavior.

In this paper we study how the visual animation of a self-avatar can be artificially modified in real-time in order to generate different haptic perceptions. In our experimental setup, participants could watch their self-avatar in a virtual environment in mirror mode while performing a weight lifting task. Users could map their gestures on the self-animated avatar in real-time using a Kinect. We introduce three kinds of modification of the visual animation of the self-avatar according to the effort delivered by the virtual avatar: 1) changes on the spatial mapping between the user’s gestures and the avatar, 2) different motion profiles of the animation, and 3) changes in the posture of the avatar (upper-body inclination). The experimental task consisted of a weight lifting task in which participants had to order four virtual dumbbells according to their virtual weight. The user had to lift each virtual dumbbells by means of a tangible stick, the animation of the avatar was modulated according to the virtual weight of the dumbbell. The results showed that the altering the spatial mapping delivered the best performance. Nevertheless, participants globally appreciated all the different visual effects. Our results pave the way to the exploitation of such novel techniques in various VR applications such as sport training, exercise games, or industrial training scenarios in single or collaborative mode.

Postural control is a dynamical process that has been extensively studied in motor control research. Recent experimental work shows a direct impact of affects on human balance. However, few studies on the automatic recognition of affects in full body expressions consider balance variables such as center of gravity displacements. Force plates enable the capture of balance variables with high precision. Automatic video extraction of the center of gravity is a basic alternative, which can be easily accessible for a wide range of public applications. This paper presents a comparison of balance variables extracted from the force plate and video processing. These variables are used to capture the bodily expressions of participants in a public speaking task designed to elicit stress. Results show that the variability of the center of gravity displacements from the force plate and video are related to negative emotions and situation appraisals. The power spectrum density broadness of the center of pressure from the force plate is related to Difficulty Describing Feelings, an important factor from a dispositional trait of Alexithymia. Implications of the use of such methods are discussed.

Databases of spontaneous multimodal expressions of affective states occurring during a task are few. This paper presents a protocol for eliciting stress in a public speaking task. Behaviors of 19 participants were recorded via a multimodal setup including speech, video of the facial expressions and body movements, balance via a force plate, and physiological measures. Questionnaires were used to assert emotional states, personality profiles and relevant coping behaviors to study how participants cope with stressful situations. Several subjective and objective performances were also evaluated. Results show a significant impact of the overall task and conditions on the participants' emotional activation. The possible future use of this new multimodal emotional corpus is described.

Analysis of non-verbal behaviors in HCI allows understanding how individuals apprehend and adapt to different situations of interaction. This seems particularly relevant when considering tasks such as speaking in a foreign language, which is known to elicit anxiety. This is even truer for young users for whom negative pedagogical feedbacks might have a strong negative impact on their motivation to learn.

We introduce the Elastic Images, a novel pseudo-haptic feedback technique which enables the perception of the local elasticity of images without the need of any haptic device. The proposed approach focus on whether visual feedback is able to induce a sensation of stiffness when the user interacts with an image using a standard mouse. The user, when clicking on a Elastic Image, is able to deform it locally according to its elastic properties. To reinforce the effect, we also propose the generation of procedural shadows and creases to simulate the compressibility of the image and several mouse cursors replacements to enhance pressure and stiffness perception. A psychophysical experiment was conducted to quantify this novel pseudo-haptic perception and determine its perceptual threshold (or its Just Noticeable Difference). The results showed that users were able to recognize up to eight different stiffness values with our proposed method and confirmed that it provides a perceivable and exploitable sensation of elasticity. The potential applications of the proposed approach range from pressure sensing in product catalogs and games, or its usage in graphical user interfaces for increasing the expressiveness of widgets.

Particle filtering is known as a robust approach for motion tracking by vision, at the cost of heavy computation in a high dimensional pose space. In this work, we describe a number of heuristics that we demonstrate to jointly improve robustness and real-time for motion capture. 3D human motion capture by monocular vision without markers can be achieved in realtime by registering a 3D articulated model on a video. First, we search the high-dimensional space of 3D poses by generating new hypotheses (or particles) with equivalent 2D projection by kinematic flipping. Second, we use a semi-deterministic particle prediction based on local optimization. Third, we deterministi-cally resample the probability distribution for a more efficient selection of particles. Particles (or poses) are evaluated using a match cost function and penalized with a Gaussian probability pose distribution learned off-line. In order to achieve real-time, measurement step is parallelized on GPU using the OpenCL API. We present experimental results demonstrating robust real-time 3D motion capture with a consumer computer and webcam.