In this work, we investigate nonverbal leadership and address two research questions: 1) is it possible to perceive leadership from nonverbal cues in an unstructured joint full-body activity with no designated leader? 2) what are its nonverbal indicators? To address these questions, we propose eight cues of nonverbal leadership and conduct a two-step validation study on a novel dataset (video, MoCap) of dance improvisation. To explore various leadership strategies, we introduce constraints on how dancers communicate by manipulating their shared sensory channels. In the first stage, 27 persons carried out continuous annotation of leadership in the recorded videos; in the second stage, 92 persons watched 25 short segments indicating who the leader was and reported perceived leadership cues. The results indicate 1) a high consensus among observers regarding nonverbal leadership, but only for certain video segments, and 2) that five leadership cues were frequently observed in our dataset. In the final part, we explore the feasibility of automatically detecting nonverbal leadership using hand-crafted cues and standard machine learning techniques.

Promoting physical activity is crucial for well-being and mental health, especially for older individuals with reduced risks of physical and psychological diseases like cognitive decline. In Italy, 17.5% of people aged 65-74 engage in sports, with a higher prevalence among men (21.6%) than women (13.7%). Theoretical models like self-determination theory (SDT) offer insights into motivational factors. SDT suggests a continuum of motivation, from absence to intrinsic motivation, with controlled and autonomous motivations driven by internal or external factors. Supporting autonomy in meeting psychological needs enhances motivation. We describe a proof-of-concept of the Age-It Project. Aiming to evaluate the impact of encouragement messages formulated based on Self-Determination Theory (SDT) received during physical activity while being immersed in an Augmented Reality (AR) Environment. The AR app acts as a virtual coach, leveraging psychological determinants to promote healthy behaviors. The objective is to foster behavioral change, enhance mental and physical functioning, and assess intervention effectiveness.

We describe a proof-of-concept of VITA, a two-year project started in the fall of 2023, aiming to develop innovative dual cognitive/motor training for healthy older adults in an Augmented Reality (AR) environment intended to foster behavioral change processes, promote general cognitive and physical functioning, and social interactions. Previous work, in fact, always focuses on individual, either cognitive or motor, training. In the paper, the project’s initial work is presented, consisting of an AR environment in which users can perform motor tasks, such as catching balloons with a ring and avoiding falling rocks, simultaneously with cognitive training.

Several attempts have been made to enhance mindfulness through Virtual and Mixed Reality. To date, they only offer users an alternative way of presenting guided imagery (e.g., mentally visualizing a beach vs. rendering a beach). We propose a preliminary study investigating whether allowing users to actively explore guided imagery through their actions (e.g., grasping virtual objects with hands) affects the mindfulness experience. To this aim, we present a preliminary study on a VR scenario for mindfulness practice that encourages the user’s interactive behavior in two conditions: interactive multimodal VR vs. audio-only. No significant difference was observed in self-reported mindfulness between the two conditions.

In this pilot study, we used a balanced A/B testing method to evaluate how an environment—a Virtual Reality (VR) head-mounted display (HMD) versus a traditional desktop—affects the performance of 20 participants on a musical pitch-colour association test. We aimed to discern the influence of testing environments on musical pitch-colour associations. Our findings revealed no significant difference in performance between the VR HMD and the traditional desktop conditions. Previous studies highlight VR’s potential, but the results of our investigation—focusing on limited immersive and presence qualities—suggest that working in VR as opposed to a standard desktop environment has no significant influence on musical pitch-colour association test results. This outcome prompts a further investigation into other inherent VR characteristics, such as spatial audio, natural environment simulations, and interactive object manipulation, which could potentially enhance the effectiveness of auditory-visual association tests. This study is the first in a series of studies to explore how VR technologies can be exploited to augment multi-modal testing, with a fully immersive musical pitch-colour association test envisioned.

Commensality, the act of eating together, offers a rich multisensory and social experience that technology can enhance. Dining involves interactions with food, where smells, colors, sounds, and textures contribute to a multisensory experience. The table becomes a focal point for social interaction, with nonverbal cues and conversations being the indispensable ingredients of the commensal experience. The CoFI2024 workshop aims to stimulate discussion about how interactive technology can enrich dining experiences. The other aim is to build an interdisciplinary community related to commensality and human-food interaction, focusing on the role of multimodal interaction among commensal partners sharing food, being humans or artificial dining companions.

Previous work introduced the concept of artificial commensal companions, i.e., embodied agents capable of interacting with humans during meals. They are supposed to bring the benefits of eating together in settings where a human would be forced to eat alone (e.g., elderly, hospitalized patients, self-isolation, etc.). This paper presents an experiment with a virtual agent and a human eating together. We invited volunteers to bring a small meal and let them chat briefly with the agent, simulating eating behaviors during the conversation. After the experience, participants filled out a questionnaire, providing quantitative and qualitative feedback. While results are encouraging (i.e., participants showed interest in eating with an agent), further work is still needed to provide more convincing results.

This demo paper presents a Virtual Reality (VR) application designed to explore innovative methods of learning basic music theory through sound-colour cross-modal associations.Leveraging the immersive capabilities of the latest head-mounted displays, the application provides an interactive and engaging platform with significant potential for music education.The demo marks an initial step in investigating how VR's immersive features can enhance the understanding of sound-colour associations and their role in learning musical concepts, specifically intervals and melodies within the C Major scale.Two groups will participate in the study: a control group will learn using a predefined sound-colour mapping, while the experimental group will create their own personalised mappings through a pre-test in VR, which will be applied in subsequent lessons.The forthcoming study aims to provide valuable insights into the effectiveness of VR and cross-modality in facilitating music theory education.

The popularity of biometrics-based user identification has significantly increased over the last few years. User identification based on the face, fingerprints, and iris, usually achieves very high accuracy only in controlled setups and can be vulnerable to presentation attacks, spoofing, and forgeries. To overcome these issues, this work proposes a novel strategy based on a relatively less explored biometric trait, i.e., gait, collected by a smartphone accelerometer, which can be more robust to the attacks mentioned above. According to the wearable sensor-based gait recognition state-of-the-art, two main classes of approaches exist: 1) those based on machine and deep learning; 2) those exploiting hand-crafted features. While the former approaches can reach a higher accuracy, they suffer from problems like, e.g., performing poorly outside the training data, i.e., lack of generalizability. This paper proposes an algorithm based on hand-crafted features for gait recognition that can outperform the existing machine and deep learning approaches. It leverages a modified Majority Voting scheme applied to Fast Window Dynamic Time Warping, a modified version of the Dynamic Time Warping (DTW) algorithm with relaxed constraints and majority voting, to recognize gait patterns. We tested our approach named MV-FWDTW on the ZJU-gaitacc, one of the most extensive datasets for the number of subjects, but especially for the number of walks per subject and walk lengths. Results set a new state-of-the-art gait recognition rate of 98.82% in a cross-session experimental setup. We also confirm the quality of the proposed method using a subset of the OU-ISIR dataset, another large state-of-the-art benchmark with more subjects but much shorter walk signals.

There is an increasing interest in combining interactive technology with food, leading to a new research area called human-food interaction. While food experiences are increasingly benefiting from interactive technology, for example in the form of food tracking apps, 3D-printed food and projections on dining tables, a more systematic advancement of the field is hindered because, so far, there is no comprehensive articulation of the grand challenges the field is facing. To further and consolidate conversations around this topic, we invited 21 HFI experts to a 5-day seminar. The goal was to review our own and prior work to identify the grand challenges in human-food interaction. The result is an articulation of 10 grand challenges in human-food interaction across 4 categories (technology, users, design and ethics). By presenting these grand challenges, we aim to help researchers move the human-food interaction research field forward.

Othello, also known as Reversi, is a popular 2-players board game. Olivaw is an intelligent agent playing Othello. Compared to the most famous ones (such as Saio), it exploits limited resources by autonomously learning how to improve its gameplay by playing against itself. In previous occasions, Othello players reported the impression of a sort of human-likeness in how Olivaw plays. We designed and ran an experimental study to better investigate these impressions in a controlled setting. Participants were asked to watch the moves of pre-recorded Othello games played by a human expert player against either another agent (i.e., Olivaw, Saio) or another human. The identity of the opponent, the outcome of the game (i.e., whether the human expert or the opponent player won), and the color of the players (i.e., black or white, black always playing first) were manipulated. We then asked participants to evaluate the human-likeness of the opponent player. Results confirm that the outcome of the match affects the perception of human-likeness of the players.

Research focusing on social interaction often leverages data-sets, allowing annotation, analysis, and modeling of social behavior. When it comes to commensality, researchers have started working on computational models of food and eating-related activities recognition. The growing research area known as Digital Commensality, has focused on meals shared online, for instance, through videochat. However, to investigate this topic, traditional data-sets recorded in laboratory settings may not be the best option in terms of ecological validity. Covid-19 restrictions and lock-downs have increased in online gatherings, with many people becoming used to the idea of sharing meals online. Following this trend, we propose the concept of collecting data by recording online interactions and discuss the challenges related to this methodology. We illustrate our approach in creating the first Digital Commensality data-set, containing recordings of food-related social interactions collected online during the Covid-19 outbreak.

Eating meals together is one of the most frequent human social experiences. When eating in the company of others, we talk, joke, laugh, and celebrate. In the paper, we focus on commensal activities, i.e., the actions related to food consumption (e.g., food chewing, in-taking) and the social signals (e.g., smiling, speaking, gazing) that appear during shared meals. We analyze the social interactions in a commensal setting and provide a baseline model for automatically recognizing such commensal activities from video recordings. More in detail, starting from a video dataset containing pairs of individuals having a meal remotely using a video-conferencing tool, we manually annotate commensal activities. We also compute several metrics, such as the number of reciprocal smiles, mutual gazes, etc., to estimate the quality of social interactions in this dataset. Next, we extract the participants’ facial activity information, and we use it to train standard classifiers (Support Vector Machines and Random Forests). Four activities are classified: chewing, speaking, food in-taking, and smiling. We apply our approach to more than 3 hours of videos collected from 18 subjects. We conclude the paper by discussing possible applications of this research in the field of Human-Agent Interaction.

Olivaw is an AI Othello playing agent which autonomously learns how to improve its gameplay by playing against itself. Some top-notch players (including former World Champions) reported that they had the impression that Olivaw's gameplay was human-like. To better investigate the processes related to these impressions, we conducted a pilot study using the Othello Game Evaluation App, a computer application we developed to evaluate pre-recorded Othello games in a controlled setting while assuring an adequate user experience. An exploratory analysis of the results shows that the participants mostly evaluated Olivaw as a human. When asked for a motivation for their choice, some of them reported that they evaluate poor game moves (and, consequently, losing the game) as an indication of the human-likeness of the player.

With one-person households being increasingly common and Covid-19 lockdown policies forcing people to stay home, remote dining has become common practice for many, who take it as an opportunity to connect with others in times of loneliness. Sharing meals online, also known as digital commensality, is a rich form of interaction, where people leverage technology to achieve a sense of connectedness and belonging while eating. In this paper, we look at digital commensality and we explore its inherent playful potential with the aim to inspire the design of engaging technologies that can support, enhance and augment this form of interaction. For this, we used a situated play design approach to document and analyze the behavior of 36 people (including pairs of friends and strangers) sharing meals online. Our analysis surfaced a set of play potentials of remote dining -- i.e., playful things people already do and enjoy spontaneously while sharing meals online. We present those play potentials as inspirational material: they can motivate and enrich the design of future digital commensality technologies by responding to people's desire for playful and social interaction with, through, and around food.

In a joint action, a group of individuals coordinate their movements to reach a shared goal. When a change–i.e., an event that affects group functioning–occurs, the group adopts strategies to face it. This article investigates how a change involving a strategic core role in a group affects interpersonal coordination, and ultimately group effectiveness in performing a joint action. Following the entrainment theory, interpersonal coordination is addressed in terms of the rhythmic cycles of the individuals and of the group and their adjustment. Music is used as an ideal ecological scenario for investigation. More specifically, this article focuses on orchestra playing. By adopting a computational approach, research is devoted to measure how a change of conductor (i.e., the leader) influences entrainment between players and its variation over time as well as the relationship between entrainment and external ratings of the orchestra performance. Results show that, whereas the change of conductor had a limited significant effect on entrainment, a significant effect was found when entrainment is used as a predictor of the external ratings. Both the obtained results and the techniques developed for measuring entrainment may open novel research directions in the area of automated analysis of group behavior, and particularly of emotion in groups.

Transactive Memory System (TMS) is a mental representation of the distribution of knowledge between the members of a team. Can an Embodied Conversational Agent perform as well as a Human when intervening as a leader to support the development of the team’s TMS? And, if yes, are there differences in the way the team perceives their respective interventions? In this paper, a perceptive online study is conducted on how Human leader interventions affect the perception of a team’s TMS. The results are compared to the ones from a previous study evaluating an Embodied Conversational agent leader rather than a human one. Both the agent and the human adopt nonverbal behaviors characterizing 2 leadership styles: Transformational (TFL) and Transactional (TAL). TFL is expected to stimulate team members curiosity and creativity in problem-solving; instead, TAL emphasizes the role of the leader in supervising the team, providing it with feedback when needed. The results show that the intervention from both the agent and the human are perceived to potentially improve the perceived TMS of a team. Another interesting insight is that the TFL style works better when performed by the Human, where both the TAL and TFL style perform well when realized by the agent.

The term commensality refers to "sharing food and eating together in a social group. In this paper, we hypothesize that it would be possible to have the same kind of experience in a HCI setting, thanks to a new type of interface that we call Artificial Commensal Companion (ACC), that would be beneficial, for example, to people who voluntarily choose or are constrained to eat alone. To this aim, we introduce an interactive system implementing an ACC in the form of a robot with non-verbal socio-affective capabilities. Future tests are already planned to evaluate its influence on the eating experience of human participants.

We present WoNoWa, a novel multi-modal dataset of small group interactions in collaborative tasks. The dataset is explicitly designed to elicit and to study over time a Transactive Memory System (TMS), a group's emergent state characterizing the group's meta-knowledge about "who knows what". A rich set of automatic features and manual annotations, extracted from the collected audio-visual data, is available on request for research purposes. Features include individual descriptors (e.g., position, Quantity of Motion, speech activity) and group descriptors (e.g., F-formations). Additionally, participants' self-assessments are available. Preliminary results from exploratory analyses show that the WoNoWa design allowed groups to develop a TMS that increased across the tasks. These results encourage the use of the WoNoWa dataset for a better understanding of the relationship between behavioural patterns and TMS, that in turn could help to improve group performance.

Game flow represents a state where the player is neither frustrated nor bored. In turn-based battle video games it can be achieved by Dynamic Difficulty Adjustment (DDA), whose research has begun rising over the last decade. This paper introduces an idea for incorporating DDA through the use of Reinforcement Learning (RL) to agents of turn-based battle video games. We design and implement an RL agent that shows, in a simple environment, the idea of how a game could achieve balance through adequate choices in actions depending on the player's level of skill.

Existing works on interactive sonification of movements, i.e., the translation of human movement qualities from the physical to the auditory domain, usually adopt a predetermined approach: the way in which movement features modulate the characteristics of sound is fixed. In our work we want to go one step further and demonstrate that the user role can influence the tuning of the mapping between movement cues and sound parameters. Here, we aim to verify if and how the mapping changes when the user is either the performer or the observer of a series of body movements (tracing a square or an infinite shape with the hand in the air). We asked participants to tune movement sonification while they were directly performing the sonified movement vs. while watching another person performing the movement and listening to its sonification. Results show that the tuning of the sonification chosen by participants is influenced by three variables: role of the user (performer vs observer), movement quality (the amount of Smoothness and Directness in the movement), and physical parameters of the movements (velocity and acceleration). Performers focused more on the quality of their movement, while observers focused more on the sonic rendering, making it more expressive and more connected to low-level physical features.

This paper presents a computational model for managing an Embodied Conversational Agent's first impressions of warmth and competence towards the user. These impressions are important to manage because they can impact users' perception of the agent and their willingness to continue the interaction with the agent. The model aims at detecting user's impression of the agent and producing appropriate agent's verbal and nonverbal behaviours in order to maintain a positive impression of warmth and competence. User's impressions are recognized using a machine learning approach with facial expressions (action units) which are important indicators of users' affective states and intentions. The agent adapts in real-time its verbal and nonverbal behaviour, with a reinforcement learning algorithm that takes user's impressions as reward to select the most appropriate combination of verbal and non-verbal behaviour to perform. A user study to test the model in a contextualized interaction with users is also presented. Our hypotheses are that users' ratings differs when the agents adapts its behaviour according to our reinforcement learning algorithm, compared to when the agent does not adapt its behaviour to user's reactions (i.e., when it randomly selects its behaviours). The study shows a general tendency for the agent to perform better when using our model than in the random condition. Significant results shows that user's ratings about agent's warmth are influenced by their a-priori about virtual characters, as well as that users' judged the agent as more competent when it adapted its behaviour compared to random condition.

Can cross-modal associations between notes and colours and between notes and emotions be strengthened by using an ad-hoc designed computer program? In this paper we present an experiment that we conducted to see (H1) if participants' associations between the auditory and visual senses could be strengthened through immersion in a multi-sensory program and (H2) if there was any correlation between auditory-visual and the auditory-emotion associations. In our experiment, participants were asked to use the SoundStrokes program, an application that translates notes played on an analog music instrument (i.e., a guitar) into coloured graphical shapes (i.e., rectangles). Between several training sessions with the program, participants were asked to associate single music notes to colours and emotions. Applications of our work may vary from entertainment, to enrich in a systematic manner the user emotional experience in, for example, videogames or immersive experiences, to digital health, in which relatively simple tools can be developed, for example, to reduce anxiety through music.

When interacting with others, we form an impression that can be declined along the two psychological dimensions of warmth and competence. By managing them, high level of engagement in an interaction can be maintained and reinforced. Our aim is to develop a virtual agent that can form and maintain a positive impression on the user that can help in improving the quality of the interaction and the user's experience. In this paper, we present an interactive system in which a virtual agent adopts a dynamic communication strategy during the interaction with a user, aiming at forming and maintaining a positive impression of warmth and competence. The agent continuously analyzes user's non-verbal signals to determine user's engagement level and adapts its communication strategy accordingly. We present a study in which we manipulate the communication strategy of the agent and we measure user's experience and user's perception of the agent's warmth and competence.

The rising popularity of learning techniques in data analysis has recently led to an increased need of large-scale datasets. In this study, we propose a system consisting of a VR game and a software platform designed to collect the player’s multimodal data, synchronized with the VR content, with the aim of creating a dataset for emotion detection and recognition. The game was implemented ad-hoc in order to elicit joy and frustration, following the emotion elicitation process described by Roseman’s appraisal theory. In this preliminary study, 5 participants played our VR game along with pre-existing ones and self-reported experienced emotions.

Creative joint activity is a form of real-time dynamic problem solving in which people collaborate to reach a common creative goal (e.g., to solve a mathematical problem, to improvise a piece of music, to write a novel, to sketch a story, and so on). While there exist interfaces able to produce social, emotional, communicative signals while collaborating with single human users to go through the creative process, the design of embodied interfaces able to observe and simultaneously effectively support creative joint activity with multiple human users is still an emerging research field. We define Creative Embodied Interfaces (CEIs) such interfaces, having either anthropomorphic or non-anthropomorphic aspect, and being either physically or virtually present in the real world. We argue that CEIs will enable a novel interaction paradigm that could be exploited in several fields such as science, education, health-care, arts, entertainment, social inclusion, companionship. This paper is aimed at providing definition and a first framework of CEIs combining psychological theories of creativity and computational models of social signal analysis/synthesis in avatars.

In this paper we present a low-intrusive approach to the detection of expressive full-body movement qualities. We focus on two qualities: Lightness and Fragility and we detect them using the data captured by four wearable devices, two Inertial Movement Units (IMU) and two electromyographs (EMG), placed on the forearms. The work we present in the paper stems from a strict collaboration with expressive movement experts (e.g., contemporary dance choreographers) for defining a vocabulary of basic movement qualities. We recorded 13 dancers performing movements expressing the qualities under investigation. The recordings were next segmented and the perceived level of each quality for each segment was ranked by 5 experts using a 5-points Likert scale. We obtained a dataset of 150 segments of movement expressing Fragility and/or Lightness. In the second part of the paper, we define a set of features on IMU and EMG data and we extract them on the recorded corpus. We finally applied a set of supervised machine learning techniques to classify the segments. The best results for the whole dataset were obtained with a Naive Bayes classifier for Lightness (F-score 0.77), and with a Support Vector Machine classifier for Fragility (F-score 0.77). Our approach can be used in ecological contexts e.g., during artistic performances.

The papers in this special section focus on the concept of laughter computing. Laughter is considered a significant feature of human-human communication. Laughter is characterized by a complex behavior that includes major modules: auditory, facial expressions, body movements, and postural attitudes, and physiological signals. The goal of this special section is to gather recent achievements in laughter computing in order to trigger new research directions in this area.

Laughter is a social signal capable of facilitating interaction in groups of people: it communicates interest, helps to improve creativity, and facilitates sociability. This article focuses on: endowing virtual characters with computational models of laughter synthesis, based on an expressivity-copying paradigm; evaluating how the physically co-presence of the laughing character impacts on the user’s perception of an audio stimulus and mood. We adopt music as a means to stimulate laughter. Results show that the character presence influences the user’s perception of music and mood. Expressivity-copying has an influence on the user’s perception of music, but does not have any significant impact on mood.

Intrapersonal synchronization of limb movements is a relevant feature for assessing coordination of motoric behavior. In this paper, we show that it can also distinguish between full-body movements performed with different expressive qualities, namely rigidity, fluidity, and impulsivity. For this purpose, we collected a dataset of movements performed by professional dancers, and annotated the perceived movement qualities with the help of a group of experts in expressive movement analysis. We computed intra personal synchronization by applying the Event Synchronization algorithm to the time-series of the speed of arms and hands. Results show that movements performed with different qualities display a significantly different amount of intra personal synchronization: impulsive movements are the most synchronized, the fluid ones show the lowest values of synchronization, and the rigid ones lay in between.

In this paper, we investigate the detection of laughter from the user's nonverbal full-body movement in social and ecological contexts. Eight hundred and one laughter and nonlaughter segments of full-body movement were examined from a corpus of motion capture data of subjects participating in social activities that stimulated laughter. A set of 13 full-body movement features was identified, and corresponding automated extraction algorithms were developed. These features were extracted from the laughter and nonlaughter segments, and the resulting dataset was provided as input to supervised machine learning techniques. Both discriminative (radial basis function-support vector machines, k-nearest neighbor, and random forest) and probabilistic (naive Bayes and logistic regression) classifiers were trained and evaluated. A comparison of automated classification with the ratings of human observers for the same laughter and nonlaughter segments showed that the performance of our approach for automated laughter detection is comparable with that of humans. The highest F-score (0.74) was obtained by the random forest classifier, whereas the F-score obtained by human observers was 0.70. Based on the analysis techniques introduced in the paper, a vision-based system prototype for automated laughter detection was designed and evaluated. Support vector machines (SVMs) and Kohonen's self-organizing maps were used for training, and the highest F-score was obtained with SVM (0.73).

In our demo, LoL, a user interacts with a virtual agentable to copy and to adapt its laughing and expressive behaviorson-the-fly. Our aim is to study copying capabilitiesparticipate in enhancing user’s experience in the interaction.User listens to funny audio stimuli in the presenceof a laughing agent: when funniness of audio increases, theagent laughs and the quality of its body movement (directionand amplitude of laughter movements) is modulated on-theflyby user’s body features.

Mimicry and laughter are two social signals displaying affiliation among people. To date, however, their relationship remains uninvestigated and relatively unexploited in designing the behaviour of robots and virtual characters. This paper presents an experiment aimed at examining how laughter and mimicry are related. The hypothesis is that hand movements a person produces during a laughter episode are mimicked through equivalent or other hand movements other participants in the interaction produce when they laugh. To investigate this, we analysed mimicry at two levels of specificity during laughter and non-laughter periods in a playful triadic social interaction. Changes in mimicry rates over the whole interaction were analysed as well as possible leader-follower relationships. Results show that hand movement rates were varied and strongly dependent on group. Even though hand movement are more frequent during laughter, mimicry does not increase. Mimicry levels, however, increase over the course of a session indicating that familiarity and comfort may increase emotional contagion.

In this paper, we study perception of intensity in-congruence between auditory and visual modalities of synthesized expressions of laughter. In particular, we investigate whether incongruent expressions are perceived as 1) regulated, and 2) unsuccessful in terms of animation synthesis. For this purpose, we conducted a perceptive study with the use of a virtual agent. Congruent and incongruent multimodal expressions of laughter were synthesized from natural audiovisual laughter episodes, using machine learning algorithms. Next, the intensity of facial expressions and body movements were systematically manipulated to check whether the resulting incongruent expressions are perceived differently compared to the corresponding congruent expressions. Results show that 1) intensity incongruence lowers the perception of believability and plausibility, and 2) the in-congruent laughter expressions displaying high intensity in the audio modality and low intensity in the body movement and facial expression are perceived as more fake than the corresponding congruent expressions. Such results have implications for both animation synthesis as well as expression regulation research.

In this paper we focus on three aspects of multimodal expressions of laughter. First, we propose a procedural method to synthesize rhythmic body movements of laughter based on spectral analysis of laughter episodes. For this purpose, we analyze laughter body motions from motion capture data and we reconstruct them with appropriate harmonics. Then we reduce the parameter space to two dimensions. These are the inputs of the actual model to generate a continuum of laughs rhythmic body movements.

When people perform a task as part of a joint action, their behavior is not the same as it would be if they were performing the same task alone: it is adapted to facilitate shared understanding (or sometimes to prevent it). Joint performance of music offers a test bed for ecologically valid investigations of the way non-verbal behavior facilitates joint action. Here we compare the expressive of violinists when playing solo Vs. in the string quartet music ensemble. The first and second violinists of a famous concert string quartet were asked to play the same musical fragment in a solo condition and with the quartet. Synchronized multimodal recordings have been created from the performances, using a specially developed software platform. The differences are not obvious to untrained observers but they are discriminated by musicians, and appropriate measures show that they exist. In particular, using an appropriate measure of entropy shows that head movements are more predictable in the quartet scenario. The change does not, as might be assumed, entail markedly reduced expression. The data pose provocative questions about joint action in realistically complex scenarios.

Within the EU ILHAIRE Project, researchers of several disciplines (e.g., computer sciences, psychology) collaborate to investigate the psychological foundations of laughter, and to bring this knowledge into shape for the use in new technologies (i.e., affective computing). Within this framework, in order to endow machines with laughter capabilities (encoding as well as decoding), one crucial task is an adequate description of laughter in terms of morphology. In this paper we present a work methodology towards automated full body laughter detection: starting from expert annotations of laughter videos we aim to identify the body features that characterize laughter.

The embodied and reflexive interaction paradigms separately proved to be effective for learning music in childhood. However, nowadays, there is a scarcity of research addressing the joined adoption of these paradigms, both from a theoretical and a technological point of view. BeSound supports children to explore - by means of their own body - rhythm, melody, and harmony and to creatively combine them together. Firstly, the child is engaged in a game in which she has to mimic the movement of the characters of stories BeSound tells her; then she can ad-lib a music dialogue with the characters. Each character was previously associated with a component of Laban's Effort and it was described through a set of whole-body movement features. These features are automatically detected, analysed, and used to control the music response of BeSound.

Embodied cooperation "arises when two co-present, individuals in motion coordinate their goal-directed actions". The adoption of the embodied cooperation paradigm for the development of embodied and social multimedia systems opens new perspectives for future User Centric Media. Systems for embodied music listening, which enable users to influence music in real-time by movement and gesture, can greatly benefit from the embodied cooperation paradigm. This paper presents the design and the evaluation of an application, Sync4All, based on such a paradigm, allowing users to experience social embodied music listening. Each user rhythmically and freely moves a mobile phone trying to synchronise her movements with those of the other ones. The level of such a synchronisation influences the music experience. The evaluation of Sync4All was aimed at finding out which is the overall attitude of the users towards the application, and how the participants perceived embodied cooperation and music embodiment.

Since its first edition in 2009, the International Workshop on Social Behaviour in Music (SBM) has been an occasion for researchers and practitioners for discussing recent advances in automated analysis of social behaviour, being music the selected test-bed and application scenario. The first edition of SBM was held in Vancouver, Canada, in the framework of the 2009 IEEE International Conference on Social Computing (SocialCom 2009). The second one was held in Genova, Italy, in the framework of the 4th International ICST Conference on Intelligent Technologies for Interactive Entertainment (Intetain 2011). SBM is now at its third edition, which takes place in the framework of the 14th International Conference on Multimodal Interaction (ICMI 2012), Santa Monica, California, USA. Again, SBM aims at providing a picture of current research breakthrough and issues, giving at the same time directions for future works and collaborations.

Successful human interaction commonly involves prototypical exchanges where interactors are engaged, synchronized, and harmonious in their behaviors. The copying of aspects of the other's behavior, at different levels, seems central to establishing and maintaining such empathic connections. Yet, many questions remain unanswered, particularly how it is possible to reflect the same affective content back to the other when the actual motion itself is not exactly the same as theirs. This paper presents a perceptual study in which emotional gestures conducted by an actor were mapped onto synthesized versions generated by an embodied virtual agent. Copying is at the expressive level, where qualities such as the fluidity or expansiveness of gestures are considered, rather than exact low-level motion matching. Participants were later asked to rate the emotional content of video recordings of both the original and the synthesized gestures. A statistical analysis shows that, in most cases, participants associated the emotional content of the agent's gestures with that intended to be expressed by the original actor. The results suggest that a combination of the type of movement performed and its quality is important for successfully communicating emotions.

In this paper, we present a system that visualizes the expressive quality of a music performance using a virtual head. We provide a mapping through several parameter spaces: on the input side, we have elaborated a mapping between values of acoustic cues and emotion as well as expressivity parameters; on the output side, we propose a mapping between these parameters and the behaviors of the virtual head. This mapping ensures a coherency between the acoustic source and the animation of the virtual head. After presenting some background information on behavior expressivity of humans, we introduce our model of expressivity. We explain how we have elaborated the mapping between the acoustic and the behavior cues. Then, we describe the implementation of a working system that controls the behavior of a human-like head that varies depending on the emotional and acoustic characteristics of the musical execution. Finally, we present the tests we conducted to validate our mapping between the emotive content of the music performance and the expressivity parameters.

Man-machine interaction (MMI) systems that utilize multimodal information about users' current emotional state are presently at the forefront of interest of the computer vision and artificial intelligence communities. A lifelike avatar can enhance interactive applications. In this paper, we present the implementation of GretaEngine and synthesized expressions, including intermediate ones, based on MPEG-4 standard and Whissel's emotion representation.

This paper introduces Gesture Engine, an animation system that synthesizes human gesturing behaviors from augmented conversation transcripts using a database of highlevel gesture definitions. An abstract scripting language to specify hand-arm gestures is introduced that incorporates knowledge from sign language research, psycholinguistics, and traditional keyframe animation. A new planning algorithm instantiates and adjusts gestures according to communicative context and temporal constraints obtained from a speech synthesizer The system animates an MPEG-4 compliant skeleton using Body Animation Parameters.