Assessing the usability of interactive systems for preschool children poses considerable difficulties due to their restricted verbal abilities, short attention spans, and tendency to favor adult researchers. Moreover, although numerous technologies for children are intended for group interaction, typical assessment approaches are still predominantly focused on individual use. This work presents the Ape-raisal, a tangible, collaborative usability assessment tool inspired by a "monkey and bananas" analogy. The tool converts a 5-point Likert scale into a tangible, fabric-based tool where children work together to "feed" monkeys to indicate their levels of engagement and satisfaction. We carried out a pilot study involving 32 children divided into six groups, utilizing a parallel triangulation method that integrated children’s self-reported information from the Ape-raisal with a systematic researcher observation framework. Our results indicate a connection between the concrete reports (Mean = 4.06, Median = 5) and the visible behavioral signs of high engagement and usability. These findings indicate that tangible and cooperative tools can successfully enable young children to participate as co-designers, offering trustworthy feedback in a group-oriented setting.

In this work, we investigate nonverbal leadership and address two research questions: 1) is it possible to perceive leadership from nonverbal cues in an unstructured joint full-body activity with no designated leader? 2) what are its nonverbal indicators? To address these questions, we propose eight cues of nonverbal leadership and conduct a two-step validation study on a novel dataset (video, MoCap) of dance improvisation. To explore various leadership strategies, we introduce constraints on how dancers communicate by manipulating their shared sensory channels. In the first stage, 27 persons carried out continuous annotation of leadership in the recorded videos; in the second stage, 92 persons watched 25 short segments indicating who the leader was and reported perceived leadership cues. The results indicate 1) a high consensus among observers regarding nonverbal leadership, but only for certain video segments, and 2) that five leadership cues were frequently observed in our dataset. In the final part, we explore the feasibility of automatically detecting nonverbal leadership using hand-crafted cues and standard machine learning techniques.

We report on the initial design steps of an audio-visual installation to be set up in a hospital guest house for children and their caregivers. The goal of the project is twofold: on one hand we aim to design a playful experience which can support children physical activity and their communication skills. On the other, we intend to develop an automated tool to assess a few basic qualities of child-caregiver interaction by interpreting users’ upper-body motion. We focus on the early stages of the project, situating it in the context of a collaboration between two institutions: an HCI lab and a paediatric hospital. We describe our efforts for the identification of a suitable dyadic activity to be first analysed using motion capture techniques and later augmented with audio-visual feedback. We also outline the strategies identified for the evaluation of the automated measurements of embodied child-caregiver interactions. Finally, we present two basic prototypes developed to stimulate a hands-on debate within our research team on the project’s design challenges and outcomes.

This poster presents an art-inspired iterative design approach applied to the definition of two experimental scenarios for movement data collection. Scenarios are inspired by warm-up exercises dancers perform to broaden group consciousness. Research focuses on the influence an expert dancer exerts on novice dancers by propagating her own movement patterns (individual motor signatures), for stimulating the emergence of a group movement pattern (group motor signature).

Industry 5.0 rethinks the role of human operators in production processes with the final goal to promote societal well-being. To achieve such a goal, novel computational approaches reshaping human-machine collaboration are needed. This paper presents a computational framework, stemmed from Cognitive Sciences, to enable human operators and machines to share a cognitive common ground in co-working hybrid processes.

This work investigates classification of emotions from full-body movements by using a novel Convolutional Neural Network-based architecture. The model is composed of two shallow networks processing in parallel where the 8-bit RGB images obtained from time intervals of 3D-positional data are the inputs. One network performs a coarse-grained modelling in the time domain while the other one applies a fine-grained modelling. We show that combining different temporal scales into one architecture improves the classification results of a dataset composed of short excerpts of the performances of professional dancers who interpreted four affective states: anger, happiness, sadness, and insecurity. Additionally, we investigate the effect of data chunk duration, overlapping, the size of the input images and the contribution of several data augmentation strategies for our proposed method. Better recognition results were obtained when the duration of a data chunk was longer, and this was further improved by applying balanced data augmentation. Moreover, we test our method on other existing motion capture datasets and compare the results with prior art. In all of the experiments, our results surpassed the state-of-the-art approaches, showing that this method generalizes across diverse settings and contexts.

Technology is increasingly widespread in schools, but it does not always find an application that fits the needs of teachers and students. A reason for that is that stakeholders are often not sufficiently involved in the design process. This paper focuses on multisensory technologies for education and on the initial stages of a design process that involved teachers and researchers with background including computer engineering, cognitive science, and digital humanities. We asked teachers to participate in brainstorming and iterative design sessions aimed at designing educational activities for kindergarten and primary school children. These include activities for a more active attitude of children during roll call, for understanding circularity of time as well as for learning mathematical topics. Questionnaires and structured interviews were used for an initial evaluation. Results are encouraging and the next step will consist in developing the mock-ups realized into applications for use and evaluation in the classroom.

Research focusing on social interaction often leverages data-sets, allowing annotation, analysis, and modeling of social behavior. When it comes to commensality, researchers have started working on computational models of food and eating-related activities recognition. The growing research area known as Digital Commensality, has focused on meals shared online, for instance, through videochat. However, to investigate this topic, traditional data-sets recorded in laboratory settings may not be the best option in terms of ecological validity. Covid-19 restrictions and lock-downs have increased in online gatherings, with many people becoming used to the idea of sharing meals online. Following this trend, we propose the concept of collecting data by recording online interactions and discuss the challenges related to this methodology. We illustrate our approach in creating the first Digital Commensality data-set, containing recordings of food-related social interactions collected online during the Covid-19 outbreak.

For technology (like serious games) that aims to deliver interactive learning, it is important to address relevant mental experiences such as reflective thinking during problem solving. To facilitate research in this direction, we present the weDraw-1 Movement Dataset of body movement sensor data and reflective thinking labels for 26 children solving mathematical problems in unconstrained settings where the body (full or parts) was required to explore these problems. Further, we provide qualitative analysis of behaviours that observers used in identifying reflective thinking moments in these sessions. The body movement cues from our compilation informed features that led to average F1 score of 0.73 for binary classification of problem-solving episodes by reflective thinking based on Long Short-Term Memory neural networks. We further obtained 0.79 average F1 score for end-to-end classification, i.e., based on raw sensor data. Finally, the algorithms resulted in 0.64 average F1 score for subsegments of these episodes as short as 4 seconds. Overall, our results show the possibility of detecting reflective thinking moments from body movement behaviours of a child exploring mathematical concepts bodily, such as within serious game play.

In a joint action, a group of individuals coordinate their movements to reach a shared goal. When a change–i.e., an event that affects group functioning–occurs, the group adopts strategies to face it. This article investigates how a change involving a strategic core role in a group affects interpersonal coordination, and ultimately group effectiveness in performing a joint action. Following the entrainment theory, interpersonal coordination is addressed in terms of the rhythmic cycles of the individuals and of the group and their adjustment. Music is used as an ideal ecological scenario for investigation. More specifically, this article focuses on orchestra playing. By adopting a computational approach, research is devoted to measure how a change of conductor (i.e., the leader) influences entrainment between players and its variation over time as well as the relationship between entrainment and external ratings of the orchestra performance. Results show that, whereas the change of conductor had a limited significant effect on entrainment, a significant effect was found when entrainment is used as a predictor of the external ratings. Both the obtained results and the techniques developed for measuring entrainment may open novel research directions in the area of automated analysis of group behavior, and particularly of emotion in groups.

In this paper, we present and assess a novel technique for unitizing inspired by a cognitive theory on event structure perception. Unitizing indicates the process of dividing an observation into smaller units. Unitizing is often performed automatically, e.g., by selecting fixed-length windows. Although fast, such approach might result in unit boundaries being placed mid-interaction, eventually affecting observation, annotation, and labeling. We conceived a unitizing technique based on the Event Segmentation theory. In brief, changes drive the perception of boundaries between events (or units): an unexpected change in the observed situation might mean the current event ended and a new one begun. Our technique relies on observed changes for identifying unit boundaries. The first sketch of our technique was recently tested, proving it effective in overcoming the aforementioned shortcomings of fixed-window unitizing. Here, we further explore its feasibility by testing it in a different domain, i.e., solo stage performances, in order to explore the feasibility of adopting our unitizing approach across domains. Our results further support the idea of leveraging the Event Segmentation Theory for the design of an automatic technique for video unitizing.

Multimodal interfaces pose the challenge of dealing with the multi-ple interactive time-scales characterizing human behavior. To dothis, innovative models and time-adaptive technologies are needed,operating at multiple time-scales and adopting a multi-layered ap-proach. The first International Workshop on Multi-Scale MovementTechnologies, hosted virtually during the 22nd ACM InternationalConference on Multimodal Interaction, is aimed at providing re-searchers from different areas with the opportunity to discuss thistopic. This paper summarizes the activities of the workshop andthe accepted papers

In this work, a computational method is proposed to automatically investigate the perception of the origin of full-body human movement and its propagation. The method is based on a mathematical game built over a suitably defined graph structure representing the human body. The players of this game are the graph vertices, which form a subset of body joints. Since each vertex contributes to a shared goal (i.e., to the way in which a specific movement-related feature is transferred among the joints), a cooperative game-theoretical model (specifically a transferable-utility game) is adopted, which is able (via the Shapley value) to measure the relevance of the various joints in human movement when performing full-body movement analysis. The method is theoretically investigated and applied to a motion capture dataset obtained from subjects who performed expressive movements. Finally, the method is validated through an online survey, in which several dancers/nondancers participated. The results show the capability of the proposed approach to represent the evolution of the most important joint responsible for originating each dancer's movement.

This paper investigates how unitizing affects external observers' annotation of group cohesion. We compared unitizing techniques belonging to these categories: interval coding, continuous coding, and a technique inspired by a cognitive theory on event perception. We applied such techniques for sampling coding units from a set of recordings of social interactions rich in behaviors related to cohesion. Then, we compared the cohesion scores the observers assigned to each coding unit. Results show that the three techniques can lead to suitable ratings and that the technique inspired to cognitive theories leads to scores reflecting variability in cohesion better than the other ones.

When developing interactive systems for children, such as serious games in the context of educational technology, it is important to take into account and address relevant cognitive and emotional child's experiences that may influence learning outcomes. Some works were done to analyze and automatically recognize these cognitive and affective states from nonverbal expressive behaviors. However, there is a lack of knowledge about visually impaired children and their body language to convey those states during learning tasks. In this paper, we present an analysis of nonverbal expressive behaviors of both blind and low-vision children, aiming at understanding what type of body communication can be an indicator of two cognitive states: engagement and confidence. In the study we consider the data collected along the EU-ICT H2020 weDRAW Project, while children were asked to solve mathematical tasks with their body. For such a dataset, we propose a list of 31 nonverbal behaviors, annotated both by rehabilitators used to work with visually impaired children and by naive observers. In the last part of the paper, we propose a preliminary study on automatic recognition of engagement and confidence states from 2D positional data. The classification results are up to 0.71 (F-score) on a three-class classification task.

Full-body human movement is characterized by fine-grain expressive qualities that humans are easily capable of exhibiting and recognizing in others’ movement. In sports (e.g., martial arts) and performing arts (e.g., dance), the same sequence of movements can be performed in a wide range of ways characterized by different qualities, often in terms of subtle (spatial and temporal) perturbations of the movement. Even a non-expert observer can distinguish between a top-level and average performance by a dancer or martial artist. The difference is not in the performed movements--the same in both cases--but in the “quality” of their performance. In this article, we present a computational framework aimed at an automated approximate measure of movement quality in full-body physical activities. Starting from motion capture data, the framework computes low-level (e.g., a limb velocity) and high-level (e.g., synchronization between different limbs) movement features. Then, this vector of features is integrated to compute a value aimed at providing a quantitative assessment of movement quality approximating the evaluation that an external expert observer would give of the same sequence of movements. Next, a system representing a concrete implementation of the framework is proposed. Karate is adopted as a testbed. We selected two different katas (i.e., detailed choreographies of movements in karate) characterized by different overall attitudes and expressions (aggressiveness, meditation), and we asked seven athletes, having various levels of experience and age, to perform them. Motion capture data were collected from the performances and were analyzed with the system. The results of the automated analysis were compared with the scores given by 14 karate experts who rated the same performances. Results show that the movement-quality scores computed by the system and the ratings given by the human observers are highly correlated (Pearson’s correlations r = 0.84, p = 0.001 and r = 0.75, p = 0.005).

Analysis of human movement data is a core topic of many research studies in human-human and human-computer interaction. Whereas, on the one side, automated movement analysis is often based on the application of sophisticated computer science techniques (e.g., motion tracking from video recordings), on the other side the interdisciplinary nature of research in this area requires the availability of tools that can be used by researchers who may not have an advanced computer science expertise. This paper presents a system enabling users, who are not necessarily computer scientists, to perform motion tracking from a dataset of video recordings. The system - consisting of a set of (freely downloadable) tools accessible by means of user friendly graphical interfaces - was designed, developed, and tested in the context of a project for automated analysis of entrainment in ensemble music performance, following the needs and requirements of musicologists and psychologists.

In this paper we present a low-intrusive approach to the detection of expressive full-body movement qualities. We focus on two qualities: Lightness and Fragility and we detect them using the data captured by four wearable devices, two Inertial Movement Units (IMU) and two electromyographs (EMG), placed on the forearms. The work we present in the paper stems from a strict collaboration with expressive movement experts (e.g., contemporary dance choreographers) for defining a vocabulary of basic movement qualities. We recorded 13 dancers performing movements expressing the qualities under investigation. The recordings were next segmented and the perceived level of each quality for each segment was ranked by 5 experts using a 5-points Likert scale. We obtained a dataset of 150 segments of movement expressing Fragility and/or Lightness. In the second part of the paper, we define a set of features on IMU and EMG data and we extract them on the recorded corpus. We finally applied a set of supervised machine learning techniques to classify the segments. The best results for the whole dataset were obtained with a Naive Bayes classifier for Lightness (F-score 0.77), and with a Support Vector Machine classifier for Fragility (F-score 0.77). Our approach can be used in ecological contexts e.g., during artistic performances.

The International Workshop on Multimodal Interaction for Education aims at investigating how multimodal interactive systems, firmly grounded on psychophysical, psychological, and pedagogical bases, can be designed, developed, and exploited for enhancing teaching and learning processes in different learning environments, with a special focus on children in the classroom. Whilst the usage of multisensory technologies in the education area is rapidly expanding, the need for solid scientific bases, design guidelines, and appropriate procedures for evaluation is emerging. Moreover, the introduction of multimodal interactive systems in the learning environment needs to develop at the same time suitable pedagogical paradigms. This workshop aims at bringing together researchers and practitioners from different disciplines, including pedagogy, psychology, psychophysics, and computer science - with a particular focus on human-computer interaction, affective computing, and social signal processing - to discuss such challenges under a multidisciplinary perspective. The workshop is partially supported by the EU-H2020-ICT Project weDRAW (http://www.wedraw.eu).

The papers in this special section focus on the concept of laughter computing. Laughter is considered a significant feature of human-human communication. Laughter is characterized by a complex behavior that includes major modules: auditory, facial expressions, body movements, and postural attitudes, and physiological signals. The goal of this special section is to gather recent achievements in laughter computing in order to trigger new research directions in this area.

Laughter is a social signal capable of facilitating interaction in groups of people: it communicates interest, helps to improve creativity, and facilitates sociability. This article focuses on: endowing virtual characters with computational models of laughter synthesis, based on an expressivity-copying paradigm; evaluating how the physically co-presence of the laughing character impacts on the user’s perception of an audio stimulus and mood. We adopt music as a means to stimulate laughter. Results show that the character presence influences the user’s perception of music and mood. Expressivity-copying has an influence on the user’s perception of music, but does not have any significant impact on mood.

Intrapersonal synchronization of limb movements is a relevant feature for assessing coordination of motoric behavior. In this paper, we show that it can also distinguish between full-body movements performed with different expressive qualities, namely rigidity, fluidity, and impulsivity. For this purpose, we collected a dataset of movements performed by professional dancers, and annotated the perceived movement qualities with the help of a group of experts in expressive movement analysis. We computed intra personal synchronization by applying the Event Synchronization algorithm to the time-series of the speed of arms and hands. Results show that movements performed with different qualities display a significantly different amount of intra personal synchronization: impulsive movements are the most synchronized, the fluid ones show the lowest values of synchronization, and the rigid ones lay in between.

Chronic (persistent) pain (CP) affects 1 in 10 adults; clinical resources are insufficient, and anxiety about activity restricts lives. Technological aids monitor activity but lack necessary psychological support. This article proposes a new sonification framework, Go-with-the-Flow, informed by physiotherapists and people with CP. The framework proposes articulation of user-defined sonified exercise spaces (SESs) tailored to psychological needs and physical capabilities that enhance body and movement awareness to rebuild confidence in physical activity. A smartphone-based wearable device and a Kinect-based device were designed based on the framework to track movement and breathing and sonify them during physical activity. In control studies conducted to evaluate the sonification strategies, people with CP reported increased performance, motivation, awareness of movement, and relaxation with sound feedback. Home studies, a focus group, and a survey of CP patients conducted at the end of a hospital pain management session provided an in-depth understanding of how different aspects of the SESs and their calibration can facilitate self-directed rehabilitation and how the wearable version of the device can facilitate transfer of gains from exercise to feared or demanding activities in real life. We conclude by discussing the implications of our findings on the design of technology for physical rehabilitation.

In this paper, we investigate the detection of laughter from the user's nonverbal full-body movement in social and ecological contexts. Eight hundred and one laughter and nonlaughter segments of full-body movement were examined from a corpus of motion capture data of subjects participating in social activities that stimulated laughter. A set of 13 full-body movement features was identified, and corresponding automated extraction algorithms were developed. These features were extracted from the laughter and nonlaughter segments, and the resulting dataset was provided as input to supervised machine learning techniques. Both discriminative (radial basis function-support vector machines, k-nearest neighbor, and random forest) and probabilistic (naive Bayes and logistic regression) classifiers were trained and evaluated. A comparison of automated classification with the ratings of human observers for the same laughter and nonlaughter segments showed that the performance of our approach for automated laughter detection is comparable with that of humans. The highest F-score (0.74) was obtained by the random forest classifier, whereas the F-score obtained by human observers was 0.70. Based on the analysis techniques introduced in the paper, a vision-based system prototype for automated laughter detection was designed and evaluated. Support vector machines (SVMs) and Kohonen's self-organizing maps were used for training, and the highest F-score was obtained with SVM (0.73).

In our demo, LoL, a user interacts with a virtual agentable to copy and to adapt its laughing and expressive behaviorson-the-fly. Our aim is to study copying capabilitiesparticipate in enhancing user’s experience in the interaction.User listens to funny audio stimuli in the presenceof a laughing agent: when funniness of audio increases, theagent laughs and the quality of its body movement (directionand amplitude of laughter movements) is modulated on-theflyby user’s body features.

How efficiently decoding affective information when computational resources and sensor systems are limited? This paper presents a framework for analysis of affective behavior starting with a reduced amount of visual information related to human upper-body movements. The main goal is to individuate a minimal representation of emotional displays based on non-verbal gesture features. The GEMEP (Geneva multimodal emotion portrayals) corpus was used to validate this framework. Twelve emotions expressed by ten actors form the selected data set of emotion portrayals. Visual tracking of trajectories of head and hands was performed from a frontal and a lateral view. Postural/shape and dynamic expressive gesture features were identified and analyzed. A feature reduction procedure was carried out, resulting in a four-dimensional model of emotion expression, that effectively classified/grouped emotions according to their valence (positive, negative) and arousal (high, low). These results show that emotionally relevant information can be detected/measured/obtained from the dynamic qualities of gesture. The framework was implemented as software modules (plug-ins) extending the EyesWeb XMI Expressive Gesture Processing Library and was tested as a component for a multimodal search engine in collaboration with Google within the EU-ICT I-SEARCH project.

Mimicry and laughter are two social signals displaying affiliation among people. To date, however, their relationship remains uninvestigated and relatively unexploited in designing the behaviour of robots and virtual characters. This paper presents an experiment aimed at examining how laughter and mimicry are related. The hypothesis is that hand movements a person produces during a laughter episode are mimicked through equivalent or other hand movements other participants in the interaction produce when they laugh. To investigate this, we analysed mimicry at two levels of specificity during laughter and non-laughter periods in a playful triadic social interaction. Changes in mimicry rates over the whole interaction were analysed as well as possible leader-follower relationships. Results show that hand movement rates were varied and strongly dependent on group. Even though hand movement are more frequent during laughter, mimicry does not increase. Mimicry levels, however, increase over the course of a session indicating that familiarity and comfort may increase emotional contagion.

In this paper, we study perception of intensity in-congruence between auditory and visual modalities of synthesized expressions of laughter. In particular, we investigate whether incongruent expressions are perceived as 1) regulated, and 2) unsuccessful in terms of animation synthesis. For this purpose, we conducted a perceptive study with the use of a virtual agent. Congruent and incongruent multimodal expressions of laughter were synthesized from natural audiovisual laughter episodes, using machine learning algorithms. Next, the intensity of facial expressions and body movements were systematically manipulated to check whether the resulting incongruent expressions are perceived differently compared to the corresponding congruent expressions. Results show that 1) intensity incongruence lowers the perception of believability and plausibility, and 2) the in-congruent laughter expressions displaying high intensity in the audio modality and low intensity in the body movement and facial expression are perceived as more fake than the corresponding congruent expressions. Such results have implications for both animation synthesis as well as expression regulation research.

In this paper we focus on three aspects of multimodal expressions of laughter. First, we propose a procedural method to synthesize rhythmic body movements of laughter based on spectral analysis of laughter episodes. For this purpose, we analyze laughter body motions from motion capture data and we reconstruct them with appropriate harmonics. Then we reduce the parameter space to two dimensions. These are the inputs of the actual model to generate a continuum of laughs rhythmic body movements.

Within the EU ILHAIRE Project, researchers of several disciplines (e.g., computer sciences, psychology) collaborate to investigate the psychological foundations of laughter, and to bring this knowledge into shape for the use in new technologies (i.e., affective computing). Within this framework, in order to endow machines with laughter capabilities (encoding as well as decoding), one crucial task is an adequate description of laughter in terms of morphology. In this paper we present a work methodology towards automated full body laughter detection: starting from expert annotations of laughter videos we aim to identify the body features that characterize laughter.

The embodied and reflexive interaction paradigms separately proved to be effective for learning music in childhood. However, nowadays, there is a scarcity of research addressing the joined adoption of these paradigms, both from a theoretical and a technological point of view. BeSound supports children to explore - by means of their own body - rhythm, melody, and harmony and to creatively combine them together. Firstly, the child is engaged in a game in which she has to mimic the movement of the characters of stories BeSound tells her; then she can ad-lib a music dialogue with the characters. Each character was previously associated with a component of Laban's Effort and it was described through a set of whole-body movement features. These features are automatically detected, analysed, and used to control the music response of BeSound.

Embodiment and reflexive interaction proved to be effective approaches to music education in childhood. A research challenge consists of merging them. This paper presents BeSound, an application intended to support children in learning the basic elements of composition. Children explore rhythm, melody, and harmony by playing at mimicking objects or characters; the qualities of their whole-body movements are analysed in real-time according to Rudolf Laban's Theory of Effort and used to control sound. The paper focuses on the design of BeSound and describes the analysis performed to distinguish between direct and flexible movements - the Laban's Space component - and between light and heavy movements - the Laban's Weight component.

Embodied cooperation "arises when two co-present, individuals in motion coordinate their goal-directed actions". The adoption of the embodied cooperation paradigm for the development of embodied and social multimedia systems opens new perspectives for future User Centric Media. Systems for embodied music listening, which enable users to influence music in real-time by movement and gesture, can greatly benefit from the embodied cooperation paradigm. This paper presents the design and the evaluation of an application, Sync4All, based on such a paradigm, allowing users to experience social embodied music listening. Each user rhythmically and freely moves a mobile phone trying to synchronise her movements with those of the other ones. The level of such a synchronisation influences the music experience. The evaluation of Sync4All was aimed at finding out which is the overall attitude of the users towards the application, and how the participants perceived embodied cooperation and music embodiment.

Since its first edition in 2009, the International Workshop on Social Behaviour in Music (SBM) has been an occasion for researchers and practitioners for discussing recent advances in automated analysis of social behaviour, being music the selected test-bed and application scenario. The first edition of SBM was held in Vancouver, Canada, in the framework of the 2009 IEEE International Conference on Social Computing (SocialCom 2009). The second one was held in Genova, Italy, in the framework of the 4th International ICST Conference on Intelligent Technologies for Interactive Entertainment (Intetain 2011). SBM is now at its third edition, which takes place in the framework of the 14th International Conference on Multimodal Interaction (ICMI 2012), Santa Monica, California, USA. Again, SBM aims at providing a picture of current research breakthrough and issues, giving at the same time directions for future works and collaborations.

This paper presents a framework for analysis of affective behavior starting with a reduced amount of visual information related to human upper-body movements. The main goal is to individuate a minimal representation of emotional displays based on nonverbal gesture features. The GEMEP (Geneva multimodal emotion portrayals) corpus was used to validate this framework. Twelve emotions expressed by 10 actors form the selected data set of emotion portrayals. Visual tracking of trajectories of head and hands were performed from a frontal and a lateral view. Postural/shape and dynamic expressive gesture features were identified and analyzed. A feature reduction procedure was carried out, resulting in a 4D model of emotion expression that effectively classified/grouped emotions according to their valence (positive, negative) and arousal (high, low). These results show that emotionally relevant information can be detected/measured/obtained from the dynamic qualities of gesture. The framework was implemented as software modules (plug-ins) extending the EyesWeb XMI Expressive Gesture Processing Library and is going to be used in user centric, networked media applications, including future mobiles, characterized by low computational resources, and limited sensor systems.

Our research focused on ensemble musical performance, an ideal test-bed for the development of models and techniques for measuring creative social interaction in an ecologically valid framework. Starting from expressive behavioral data of a string quartet, this paper addresses the application of Multi-Scale Entropy method to investigate dominance.

This paper presents a multimodal system for real-time analysis of nonverbal affective social interaction in small groups of users. The focus is on two major aspects of affective social interaction: the synchronization of the affective behavior within a small group and the emergence of functional roles, such as leadership. A small group of users is modeled as a complex system consisting of single interacting components that can auto-organize and show global properties. Techniques are developed for computing quantitative measures of both synchronization and leadership. Music is selected as experimental test-bed since it is a clear example of interactive and social activity, where affective nonverbal communication plays a fundamental role. The system has been implemented as software modules for the EyesWeb XMI platform (http://www.eyesweb.org). It has been used in experimental frameworks (a violin duo and a string quartet) and in real-world applications (in user-centric applications for active music listening). Further application scenarios include entertainment, edutainment, therapy and rehabilitation, cultural heritage, and museum applications. Research has been carried out in the framework of the EU-ICT FP7 Project SAME (http://www.sameproject.eu).

This work aims at defining a computational model of human emotional entrainment. Music, as a non-verbal language to express emotions, is chosen as an ideal test bed for these aims. We start from multimodal gesture and motion signals, recorded in a real world collaborative condition in an ecological setting. Four violin players were asked to play, alone or in duo, a music fragment in two different perceptual feedback modalities and in four different emotional states. We focused our attention on phase synchronisation of the head motions of the players. From observation by subjects (musicians and observers), an evidence of entrainment emerges between players. The preliminary results, based on a reduced data set, however do not grasp fully this phenomenon. A more extended analysis is current subject of investigation.

This paper focuses on multimodal and cross-modal analysis of expressive gesture with a particular focus on collaborative interactive systems exploiting tangible acoustic interfaces (TAIs). We developed TAIs aiming at processing expressive information from users and supporting creativity in concrete music theatre and museum projects. The paper presents (i) techniques for extraction and analysis of high-level features from expressive gesture of TAIs users in collaborative frameworks, (ii) concrete examples of multimodal and cross-modal processing of expressive gesture, (iii) examples of how such results have been exploited in public events and artistic productions. In such occasions the developed techniques have been applied and evaluated with experiments involving both experts and the general audience. Research is carried out in the framework of the EU-IST STREP Project TAI-CHI (Tangible Acoustic Interfaces for Computer-Human Interaction). High-level expressive gesture analysis and multimodal and cross-modal processing are achieved in the new EyesWeb 4 open platform (available at www.eyesweb.org)

Abstract This paper presents and discusses some experiments having the purpose of planning, developing and validating aesthetically resonant environments for different types of sensorimotor impairments which affect Parkinson's patients. From a technical point of view the aim is to develop a computational open architecture in which it is possible to integrate modules for gesture analysis and recognition and for interactive construction of therapeutic exercises based on multimedia stimulation in real time. The clinical objective is to experiment with a device of sensorimotor stimulation that supports akinesia compensation by controlling movement rhythmic structures in subjects with Parkinson's disease. The EyesWeb open architecture has been used to analyse patients' motion and to produce visual feedback during therapy sessions in real time. A pilot study has been conducted on two Parkinson's disease patients in the framework of the EU IST project CARE‐HERE. Copyright © 2003 John Wiley & Sons, Ltd.