In this work, we investigate nonverbal leadership and address two research questions: 1) is it possible to perceive leadership from nonverbal cues in an unstructured joint full-body activity with no designated leader? 2) what are its nonverbal indicators? To address these questions, we propose eight cues of nonverbal leadership and conduct a two-step validation study on a novel dataset (video, MoCap) of dance improvisation. To explore various leadership strategies, we introduce constraints on how dancers communicate by manipulating their shared sensory channels. In the first stage, 27 persons carried out continuous annotation of leadership in the recorded videos; in the second stage, 92 persons watched 25 short segments indicating who the leader was and reported perceived leadership cues. The results indicate 1) a high consensus among observers regarding nonverbal leadership, but only for certain video segments, and 2) that five leadership cues were frequently observed in our dataset. In the final part, we explore the feasibility of automatically detecting nonverbal leadership using hand-crafted cues and standard machine learning techniques.

Decoding the intensity of facial expressions is of primary importance for humans. Modeling this computationally, however, is not an easy task. Here, we propose a new dataset composed of circa 400 videos and 1,000 images automatically extracted from several movies, and rated by humans on intensity. Each stimulus presents facial expressions of one person only, but overall, the stimuli represent a large variety of expressions in individuals of different age, gender, and ethnicity, in fictional yet natural movie settings. Each video was rated by 5 people in terms of perceived intensity and variability using a 7-point Likert scale; each image was rated by 5 people only for intensity. In total, 90 people participated in the ratings, and the average inter-rater ICC agreement is 0.63 for videos and 0.66 for images. For each video and image we also extracted action units using the OpenFace software. We report results for both human and computer-assisted intensity ratings, and propose a baseline regression model capable of estimating the perceived intensity in videos with a mean squared error of 0.74. We conclude our paper by discussing potential applications of a general computational model of perceived intensity.

Social communication involves interpreting nonverbal behaviors, detecting and anticipating others’ actions and intentions. Actions convey not only the goal and motor intention but also the form, i.e., variations in action execution. These variations, termed vitality forms, communicate attitudes during interactions, such as being gentle, calm, vigorous, and rude. Automatic vitality form recognition may have several applications in social robotics, social skills training, and therapy, yet it remains a rarely studied topic. This paper introduces an unsupervised pre-training approach that utilizes 2D-body key point trajectories as input and employs diffusion models to derive more effective features for representing these trajectories. The features learned from the diffusion model’s encoder are utilized to train a multilayer perceptron for vitality form recognition. Experimental analysis showcases the superior performance of the proposed method not only across various videos but also for action classes not encountered during training.

In this paper, we investigate the impact of sonification on the willingness for physical contact. For this purpose, we introduce a novel system designed to explore this impact within a performative art setting. It consists of a MIDI controller which detects physical contact between two dancers and transforms it into sounds. We use it in a preliminary experiment aimed at investigating whether sonification of physical contact in contact dance improvisations influences participants’ behavior and their experience. Three sonification strategies are explored with 10 participants performing improvisations with the professional dancer. Questionnaires, interviews and quantification of touches were utilized to compare experimental conditions. The results suggest that sonification of the physical contact influence participants’ behavior and experiences, although this effect may depend on type of sonification.

Several attempts have been made to enhance mindfulness through Virtual and Mixed Reality. To date, they only offer users an alternative way of presenting guided imagery (e.g., mentally visualizing a beach vs. rendering a beach). We propose a preliminary study investigating whether allowing users to actively explore guided imagery through their actions (e.g., grasping virtual objects with hands) affects the mindfulness experience. To this aim, we present a preliminary study on a VR scenario for mindfulness practice that encourages the user’s interactive behavior in two conditions: interactive multimodal VR vs. audio-only. No significant difference was observed in self-reported mindfulness between the two conditions.

We propose a novel dataset for studying and modeling facial expression intensity. Facial expression intensity recognition is a rarely discussed challenge, likely stemming from a lack of suitable datasets. Our dataset has been created by extracting facial expressions from actors across twelve fiction films, followed by crowd-sourced online annotation of the expression intensity and variability levels. It consists of over 400 automatically extracted video segments ranging from 3 to 5 seconds, as well as annotations and facial landmarks. We also present preliminary statistics derived from this dataset.

Commensality, the act of eating together, offers a rich multisensory and social experience that technology can enhance. Dining involves interactions with food, where smells, colors, sounds, and textures contribute to a multisensory experience. The table becomes a focal point for social interaction, with nonverbal cues and conversations being the indispensable ingredients of the commensal experience. The CoFI2024 workshop aims to stimulate discussion about how interactive technology can enrich dining experiences. The other aim is to build an interdisciplinary community related to commensality and human-food interaction, focusing on the role of multimodal interaction among commensal partners sharing food, being humans or artificial dining companions.

Previous work introduced the concept of artificial commensal companions, i.e., embodied agents capable of interacting with humans during meals. They are supposed to bring the benefits of eating together in settings where a human would be forced to eat alone (e.g., elderly, hospitalized patients, self-isolation, etc.). This paper presents an experiment with a virtual agent and a human eating together. We invited volunteers to bring a small meal and let them chat briefly with the agent, simulating eating behaviors during the conversation. After the experience, participants filled out a questionnaire, providing quantitative and qualitative feedback. While results are encouraging (i.e., participants showed interest in eating with an agent), further work is still needed to provide more convincing results.

This work investigates classification of emotions from full-body movements by using a novel Convolutional Neural Network-based architecture. The model is composed of two shallow networks processing in parallel where the 8-bit RGB images obtained from time intervals of 3D-positional data are the inputs. One network performs a coarse-grained modelling in the time domain while the other one applies a fine-grained modelling. We show that combining different temporal scales into one architecture improves the classification results of a dataset composed of short excerpts of the performances of professional dancers who interpreted four affective states: anger, happiness, sadness, and insecurity. Additionally, we investigate the effect of data chunk duration, overlapping, the size of the input images and the contribution of several data augmentation strategies for our proposed method. Better recognition results were obtained when the duration of a data chunk was longer, and this was further improved by applying balanced data augmentation. Moreover, we test our method on other existing motion capture datasets and compare the results with prior art. In all of the experiments, our results surpassed the state-of-the-art approaches, showing that this method generalizes across diverse settings and contexts.

Research focusing on social interaction often leverages data-sets, allowing annotation, analysis, and modeling of social behavior. When it comes to commensality, researchers have started working on computational models of food and eating-related activities recognition. The growing research area known as Digital Commensality, has focused on meals shared online, for instance, through videochat. However, to investigate this topic, traditional data-sets recorded in laboratory settings may not be the best option in terms of ecological validity. Covid-19 restrictions and lock-downs have increased in online gatherings, with many people becoming used to the idea of sharing meals online. Following this trend, we propose the concept of collecting data by recording online interactions and discuss the challenges related to this methodology. We illustrate our approach in creating the first Digital Commensality data-set, containing recordings of food-related social interactions collected online during the Covid-19 outbreak.

Eating meals together is one of the most frequent human social experiences. When eating in the company of others, we talk, joke, laugh, and celebrate. In the paper, we focus on commensal activities, i.e., the actions related to food consumption (e.g., food chewing, in-taking) and the social signals (e.g., smiling, speaking, gazing) that appear during shared meals. We analyze the social interactions in a commensal setting and provide a baseline model for automatically recognizing such commensal activities from video recordings. More in detail, starting from a video dataset containing pairs of individuals having a meal remotely using a video-conferencing tool, we manually annotate commensal activities. We also compute several metrics, such as the number of reciprocal smiles, mutual gazes, etc., to estimate the quality of social interactions in this dataset. Next, we extract the participants’ facial activity information, and we use it to train standard classifiers (Support Vector Machines and Random Forests). Four activities are classified: chewing, speaking, food in-taking, and smiling. We apply our approach to more than 3 hours of videos collected from 18 subjects. We conclude the paper by discussing possible applications of this research in the field of Human-Agent Interaction.

As social agents, we experience situations in which sometimes we enjoy being involved and others where we desire to withdraw from. Being aware of others’ “comfort towards the interaction” help us enhance our communications, thus this becomes a fundamental skill for any interactive agent (either a robot or an Embodied Conversational Agent (ECA)). For this reason, the current paper considers Comfortability, the internal state that focuses on the person’s desire to maintain or withdraw from an interaction, exploring whether it is possible to recognize it from human non-verbal behaviour. To this aim, videos collected during real Human-Robot Interactions (HRI) were segmented, manually annotated and used to train four standard classifiers. Concretely, different combinations of various facial and upper-body movements (i.e., Action Units, Head Pose, Upper-body Pose and Gaze) were fed to the following feature-based Machine Learning (ML) algorithms: Naive Bayes, Neural Networks, Random Forest and Support Vector Machines. The results indicate that the best model, obtaining a 75% recognition accuracy, is trained with all the aforementioned cues together and based on Random Forest. These findings indicate, for the first time, that Comfortability can be automatically recognized, paving the way to its future integration into interactive agents.

For technology (like serious games) that aims to deliver interactive learning, it is important to address relevant mental experiences such as reflective thinking during problem solving. To facilitate research in this direction, we present the weDraw-1 Movement Dataset of body movement sensor data and reflective thinking labels for 26 children solving mathematical problems in unconstrained settings where the body (full or parts) was required to explore these problems. Further, we provide qualitative analysis of behaviours that observers used in identifying reflective thinking moments in these sessions. The body movement cues from our compilation informed features that led to average F1 score of 0.73 for binary classification of problem-solving episodes by reflective thinking based on Long Short-Term Memory neural networks. We further obtained 0.79 average F1 score for end-to-end classification, i.e., based on raw sensor data. Finally, the algorithms resulted in 0.64 average F1 score for subsegments of these episodes as short as 4 seconds. Overall, our results show the possibility of detecting reflective thinking moments from body movement behaviours of a child exploring mathematical concepts bodily, such as within serious game play.

In this paper, we investigate whether information related to touches and rotations impressed to an object can be effectively used to classify the emotion of the agent manipulating it. We specifically focus on sequences of basic actions (e.g., grasping, rotating), which are constituents of daily interactions. We use the iCube, a 5 cm cube covered with tactile sensors and embedded with an accelometer, to collect a new dataset including 11 persons performing action sequences associated with 4 emotions: anger, sadness, excitement and gratitude. Next, we propose 17 high-level hand-crafted features based on the tactile and kinematics data derived from the iCube. Twelve of these features vary significantly as a function of the emotional context in which the action sequence was performed. In particular, a larger surface of the object is engaged in physical contact for anger and excitement, than for sadness. Furthermore, the average duration of interactions labeled as sad, is longer than for the remaining 3 emotions. More rotations are performed for anger and excitement than for sadness and gratitude. The accuracy of a classification experiment in the case of four emotions reaches 0.75. This result shows that the emotion recognition during hand-object interactions is possible and it may foster development of new intelligent user interfaces.

The term commensality refers to "sharing food and eating together in a social group. In this paper, we hypothesize that it would be possible to have the same kind of experience in a HCI setting, thanks to a new type of interface that we call Artificial Commensal Companion (ACC), that would be beneficial, for example, to people who voluntarily choose or are constrained to eat alone. To this aim, we introduce an interactive system implementing an ACC in the form of a robot with non-verbal socio-affective capabilities. Future tests are already planned to evaluate its influence on the eating experience of human participants.

When developing interactive systems for children, such as serious games in the context of educational technology, it is important to take into account and address relevant cognitive and emotional child's experiences that may influence learning outcomes. Some works were done to analyze and automatically recognize these cognitive and affective states from nonverbal expressive behaviors. However, there is a lack of knowledge about visually impaired children and their body language to convey those states during learning tasks. In this paper, we present an analysis of nonverbal expressive behaviors of both blind and low-vision children, aiming at understanding what type of body communication can be an indicator of two cognitive states: engagement and confidence. In the study we consider the data collected along the EU-ICT H2020 weDRAW Project, while children were asked to solve mathematical tasks with their body. For such a dataset, we propose a list of 31 nonverbal behaviors, annotated both by rehabilitators used to work with visually impaired children and by naive observers. In the last part of the paper, we propose a preliminary study on automatic recognition of engagement and confidence states from 2D positional data. The classification results are up to 0.71 (F-score) on a three-class classification task.

The rising popularity of learning techniques in data analysis has recently led to an increased need of large-scale datasets. In this study, we propose a system consisting of a VR game and a software platform designed to collect the player’s multimodal data, synchronized with the VR content, with the aim of creating a dataset for emotion detection and recognition. The game was implemented ad-hoc in order to elicit joy and frustration, following the emotion elicitation process described by Roseman’s appraisal theory. In this preliminary study, 5 participants played our VR game along with pre-existing ones and self-reported experienced emotions.

Full-body human movement is characterized by fine-grain expressive qualities that humans are easily capable of exhibiting and recognizing in others’ movement. In sports (e.g., martial arts) and performing arts (e.g., dance), the same sequence of movements can be performed in a wide range of ways characterized by different qualities, often in terms of subtle (spatial and temporal) perturbations of the movement. Even a non-expert observer can distinguish between a top-level and average performance by a dancer or martial artist. The difference is not in the performed movements--the same in both cases--but in the “quality” of their performance. In this article, we present a computational framework aimed at an automated approximate measure of movement quality in full-body physical activities. Starting from motion capture data, the framework computes low-level (e.g., a limb velocity) and high-level (e.g., synchronization between different limbs) movement features. Then, this vector of features is integrated to compute a value aimed at providing a quantitative assessment of movement quality approximating the evaluation that an external expert observer would give of the same sequence of movements. Next, a system representing a concrete implementation of the framework is proposed. Karate is adopted as a testbed. We selected two different katas (i.e., detailed choreographies of movements in karate) characterized by different overall attitudes and expressions (aggressiveness, meditation), and we asked seven athletes, having various levels of experience and age, to perform them. Motion capture data were collected from the performances and were analyzed with the system. The results of the automated analysis were compared with the scores given by 14 karate experts who rated the same performances. Results show that the movement-quality scores computed by the system and the ratings given by the human observers are highly correlated (Pearson’s correlations r = 0.84, p = 0.001 and r = 0.75, p = 0.005).

In this paper we present a low-intrusive approach to the detection of expressive full-body movement qualities. We focus on two qualities: Lightness and Fragility and we detect them using the data captured by four wearable devices, two Inertial Movement Units (IMU) and two electromyographs (EMG), placed on the forearms. The work we present in the paper stems from a strict collaboration with expressive movement experts (e.g., contemporary dance choreographers) for defining a vocabulary of basic movement qualities. We recorded 13 dancers performing movements expressing the qualities under investigation. The recordings were next segmented and the perceived level of each quality for each segment was ranked by 5 experts using a 5-points Likert scale. We obtained a dataset of 150 segments of movement expressing Fragility and/or Lightness. In the second part of the paper, we define a set of features on IMU and EMG data and we extract them on the recorded corpus. We finally applied a set of supervised machine learning techniques to classify the segments. The best results for the whole dataset were obtained with a Naive Bayes classifier for Lightness (F-score 0.77), and with a Support Vector Machine classifier for Fragility (F-score 0.77). Our approach can be used in ecological contexts e.g., during artistic performances.

The papers in this special section focus on the concept of laughter computing. Laughter is considered a significant feature of human-human communication. Laughter is characterized by a complex behavior that includes major modules: auditory, facial expressions, body movements, and postural attitudes, and physiological signals. The goal of this special section is to gather recent achievements in laughter computing in order to trigger new research directions in this area.

Intrapersonal synchronization of limb movements is a relevant feature for assessing coordination of motoric behavior. In this paper, we show that it can also distinguish between full-body movements performed with different expressive qualities, namely rigidity, fluidity, and impulsivity. For this purpose, we collected a dataset of movements performed by professional dancers, and annotated the perceived movement qualities with the help of a group of experts in expressive movement analysis. We computed intra personal synchronization by applying the Event Synchronization algorithm to the time-series of the speed of arms and hands. Results show that movements performed with different qualities display a significantly different amount of intra personal synchronization: impulsive movements are the most synchronized, the fluid ones show the lowest values of synchronization, and the rigid ones lay in between.

In this paper, we investigate the detection of laughter from the user's nonverbal full-body movement in social and ecological contexts. Eight hundred and one laughter and nonlaughter segments of full-body movement were examined from a corpus of motion capture data of subjects participating in social activities that stimulated laughter. A set of 13 full-body movement features was identified, and corresponding automated extraction algorithms were developed. These features were extracted from the laughter and nonlaughter segments, and the resulting dataset was provided as input to supervised machine learning techniques. Both discriminative (radial basis function-support vector machines, k-nearest neighbor, and random forest) and probabilistic (naive Bayes and logistic regression) classifiers were trained and evaluated. A comparison of automated classification with the ratings of human observers for the same laughter and nonlaughter segments showed that the performance of our approach for automated laughter detection is comparable with that of humans. The highest F-score (0.74) was obtained by the random forest classifier, whereas the F-score obtained by human observers was 0.70. Based on the analysis techniques introduced in the paper, a vision-based system prototype for automated laughter detection was designed and evaluated. Support vector machines (SVMs) and Kohonen's self-organizing maps were used for training, and the highest F-score was obtained with SVM (0.73).

In this paper, we study perception of intensity in-congruence between auditory and visual modalities of synthesized expressions of laughter. In particular, we investigate whether incongruent expressions are perceived as 1) regulated, and 2) unsuccessful in terms of animation synthesis. For this purpose, we conducted a perceptive study with the use of a virtual agent. Congruent and incongruent multimodal expressions of laughter were synthesized from natural audiovisual laughter episodes, using machine learning algorithms. Next, the intensity of facial expressions and body movements were systematically manipulated to check whether the resulting incongruent expressions are perceived differently compared to the corresponding congruent expressions. Results show that 1) intensity incongruence lowers the perception of believability and plausibility, and 2) the in-congruent laughter expressions displaying high intensity in the audio modality and low intensity in the body movement and facial expression are perceived as more fake than the corresponding congruent expressions. Such results have implications for both animation synthesis as well as expression regulation research.

This article focuses on the identification and perception of facial action units displayed alone as well as the meaning decoding and perception of full-face synthesized expressions of laughter. We argue that the adequate representation of single action units is important in the decoding and perception of full-face expressions. In particular, we focus on three factors that may influence the identification and perception of single actions and full-face expressions: their presentation mode (static vs. dynamic), their intensity, and the presence of wrinkles. For the purpose of this study, we used a hybrid approach for animation synthesis that combines data-driven and procedural animations with synthesized wrinkles generated using a bump mapping method. Using such animation technique, we created animations of single action units and full-face movements of two virtual characters. Next, we conducted two studies to evaluate the role of presentation mode, intensity, and wrinkles in single actions and full-face context-free expressions. Our evaluation results show that intensity and presentation mode influence (1) the identification of single action units and (2) the perceived quality of the animation. At the same time, wrinkles (3) are useful in the identification of a single action unit and (4) influence the perceived meaning attached to the animation of full-face expressions. Thus, all factors are important for successful communication of expressions displayed by virtual characters.

In this paper we focus on three aspects of multimodal expressions of laughter. First, we propose a procedural method to synthesize rhythmic body movements of laughter based on spectral analysis of laughter episodes. For this purpose, we analyze laughter body motions from motion capture data and we reconstruct them with appropriate harmonics. Then we reduce the parameter space to two dimensions. These are the inputs of the actual model to generate a continuum of laughs rhythmic body movements.

Emotional expressions play a very important role in the interaction between virtual agents and human users. In this paper, we present a new constraint-based approach to the generation of multimodal emotional displays. The displays generated with our method are not limited to the face, but are composed of different signals partially ordered in time and belonging to different modalities. We also describe the evaluation of the main features of our approach. We examine the role of multimodality, sequentiality, and constraints in the perception of synthesized emotional states. The results of our evaluation show that applying our algorithm improves the communication of a large spectrum of emotional states, while the believability of the agent animations increases with the use of constraints over the multimodal signals.