In this work, we investigate nonverbal leadership and address two research questions: 1) is it possible to perceive leadership from nonverbal cues in an unstructured joint full-body activity with no designated leader? 2) what are its nonverbal indicators? To address these questions, we propose eight cues of nonverbal leadership and conduct a two-step validation study on a novel dataset (video, MoCap) of dance improvisation. To explore various leadership strategies, we introduce constraints on how dancers communicate by manipulating their shared sensory channels. In the first stage, 27 persons carried out continuous annotation of leadership in the recorded videos; in the second stage, 92 persons watched 25 short segments indicating who the leader was and reported perceived leadership cues. The results indicate 1) a high consensus among observers regarding nonverbal leadership, but only for certain video segments, and 2) that five leadership cues were frequently observed in our dataset. In the final part, we explore the feasibility of automatically detecting nonverbal leadership using hand-crafted cues and standard machine learning techniques.

We report on the initial design steps of an audio-visual installation to be set up in a hospital guest house for children and their caregivers. The goal of the project is twofold: on one hand we aim to design a playful experience which can support children physical activity and their communication skills. On the other, we intend to develop an automated tool to assess a few basic qualities of child-caregiver interaction by interpreting users’ upper-body motion. We focus on the early stages of the project, situating it in the context of a collaboration between two institutions: an HCI lab and a paediatric hospital. We describe our efforts for the identification of a suitable dyadic activity to be first analysed using motion capture techniques and later augmented with audio-visual feedback. We also outline the strategies identified for the evaluation of the automated measurements of embodied child-caregiver interactions. Finally, we present two basic prototypes developed to stimulate a hands-on debate within our research team on the project’s design challenges and outcomes.

This poster presents an art-inspired iterative design approach applied to the definition of two experimental scenarios for movement data collection. Scenarios are inspired by warm-up exercises dancers perform to broaden group consciousness. Research focuses on the influence an expert dancer exerts on novice dancers by propagating her own movement patterns (individual motor signatures), for stimulating the emergence of a group movement pattern (group motor signature).

This work investigates classification of emotions from full-body movements by using a novel Convolutional Neural Network-based architecture. The model is composed of two shallow networks processing in parallel where the 8-bit RGB images obtained from time intervals of 3D-positional data are the inputs. One network performs a coarse-grained modelling in the time domain while the other one applies a fine-grained modelling. We show that combining different temporal scales into one architecture improves the classification results of a dataset composed of short excerpts of the performances of professional dancers who interpreted four affective states: anger, happiness, sadness, and insecurity. Additionally, we investigate the effect of data chunk duration, overlapping, the size of the input images and the contribution of several data augmentation strategies for our proposed method. Better recognition results were obtained when the duration of a data chunk was longer, and this was further improved by applying balanced data augmentation. Moreover, we test our method on other existing motion capture datasets and compare the results with prior art. In all of the experiments, our results surpassed the state-of-the-art approaches, showing that this method generalizes across diverse settings and contexts.

The execution of a human movement involves different muscles that are activated and coordinated by the brain at different temporal scales in a complex cognitive process. For this reason, studying human motion requires to properly model multiple temporal scales that fully describe its complexity. Current approaches are not able to address this requirement properly or are based on oversimplified models with obvious limitations. Data-driven models represent research frontiers able to provide new insights. In this work we will investigate different data-driven approaches. The first one is based on shallow models that, while achieving reasonably good recognition performance, require to handcraft features according to the domain knowledge The second one is based on deep models that can be extended to manage multiple temporal scales but they are hard to exploit as too many architecture configurations exist. For this reason, we will propose a new deep multiple temporal scale data-driven model, based on Temporal Convolutional Network, capable of learning features from the data at different temporal scales, of outperforming state of the art deep and shallow models, and of exploiting shallow models to tune the architecture configuration. We designed, collected data and tested our proposal in a specially devised experiment, to prove the validity of our approach. In particular, we collected motion capture data about dyad actions where two people exchange a ball. As the weight of the ball and the throwing intentions change, we will show how it is possible to automatically detect either the weight of the ball or the intention behind the throw just based on motion data. Data regarding our experiment and code of the methods proposed in this work are also made freely available to the research community. Results support both the proposal and the need for the use of deep multi scale models as a tool to better understand human movement and its multiple time scale nature.

Studying human motion requires modelling its multiple temporal scale nature to fully describe its complexity since different muscles are activated and coordinated by the brain at different temporal scales in a complex cognitive process. Nevertheless, current approaches are not able to address this requirement properly, and are based on oversimplified models with obvious limitations. Data-driven methods represent a viable tool to address these limitations. Nevertheless, shallow data-driven models, while achieving reasonably good recognition performance, require to handcraft features based on domain-specific knowledge which, in this cases, is limited and does no allow to properly model motion- and subject-specific temporal scales. In this work, we propose a new deep multiple temporal scale data-driven model, based on Temporal Convolutional Networks, able to automatically learn features from the data at different temporal scales. Our proposal focuses first on over-performing state-of-the-art shallows and deep models in terms of recognition performance. Then, thanks to the use of feature ranking for shallow models and an attention map for deep models, we will give insights on what the different architectures actually learned from the data. We designed, collected data, and tested our proposal in custom experiment of motion recognition: detecting the person who draw a particular shape (i.e., an ellipse) on a graphics tablet, collecting data about his/her movement (e.g., pressure and speed) in different extrapolating scenarios (e.g., training with data collected from one hand and testing the model on the other one). Collected data regarding our experiment and code of the methods are also made freely available to the research community. Results, both in terms of accuracy and insight on the cognitive problem, support the proposal and support the use of the proposed technique as a support tool for better understanding the human movements and its multiple temporal scale nature.

In a joint action, a group of individuals coordinate their movements to reach a shared goal. When a change–i.e., an event that affects group functioning–occurs, the group adopts strategies to face it. This article investigates how a change involving a strategic core role in a group affects interpersonal coordination, and ultimately group effectiveness in performing a joint action. Following the entrainment theory, interpersonal coordination is addressed in terms of the rhythmic cycles of the individuals and of the group and their adjustment. Music is used as an ideal ecological scenario for investigation. More specifically, this article focuses on orchestra playing. By adopting a computational approach, research is devoted to measure how a change of conductor (i.e., the leader) influences entrainment between players and its variation over time as well as the relationship between entrainment and external ratings of the orchestra performance. Results show that, whereas the change of conductor had a limited significant effect on entrainment, a significant effect was found when entrainment is used as a predictor of the external ratings. Both the obtained results and the techniques developed for measuring entrainment may open novel research directions in the area of automated analysis of group behavior, and particularly of emotion in groups.

An evaluation study of a serious game and a system for the automatic emotion recognition designed for helping autistic children to learn to recognize and express emotions by means of their full-body movement is presented. Three-dimensional motion data of full-body movements are obtained from RGB-D sensors and used to recognize emotions by means of linear SVMs. Ten children diagnosed with High Functioning Autism or Asperger Syndrome were involved in the evaluation phase, consisting of repeated sessions to play a specifically designed serious game. Results from the evaluation study show an increase of tasks accuracy from the beginning to the end of training sessions in the trained group. In particular, while the increase of recognition accuracy was concentrated in the first sessions of the game, the increase for expression accuracy is more gradual throughout all sessions. Moreover, the training seems to produce a transfer effect on facial expression recognition.

Multimodal interfaces pose the challenge of dealing with the multi-ple interactive time-scales characterizing human behavior. To dothis, innovative models and time-adaptive technologies are needed,operating at multiple time-scales and adopting a multi-layered ap-proach. The first International Workshop on Multi-Scale MovementTechnologies, hosted virtually during the 22nd ACM InternationalConference on Multimodal Interaction, is aimed at providing re-searchers from different areas with the opportunity to discuss thistopic. This paper summarizes the activities of the workshop andthe accepted papers

In this work, a computational method is proposed to automatically investigate the perception of the origin of full-body human movement and its propagation. The method is based on a mathematical game built over a suitably defined graph structure representing the human body. The players of this game are the graph vertices, which form a subset of body joints. Since each vertex contributes to a shared goal (i.e., to the way in which a specific movement-related feature is transferred among the joints), a cooperative game-theoretical model (specifically a transferable-utility game) is adopted, which is able (via the Shapley value) to measure the relevance of the various joints in human movement when performing full-body movement analysis. The method is theoretically investigated and applied to a motion capture dataset obtained from subjects who performed expressive movements. Finally, the method is validated through an online survey, in which several dancers/nondancers participated. The results show the capability of the proposed approach to represent the evolution of the most important joint responsible for originating each dancer's movement.

“Serious games” are becoming extremely relevant to individuals who have specific needs, such as children with an autism spectrum condition (ASC). Often, individuals with an ASC have difficulties in interpreting verbal and nonverbal communication cues during social interactions. The ASC-Inclusion EU-FP7 funded project aims to provide children who have an ASC with a platform to learn emotion expression and recognition, through play in the virtual world. In particular, the ASC-Inclusion platform focuses on the expression of emotion via facial, vocal, and bodily gestures. The platform combines multiple analysis tools, using onboard microphone and webcam capabilities. The platform utilizes these capabilities via training games, text-based communication, animations, video, and audio clips. This paper introduces current findings and evaluations of the ASC-Inclusion platform and provides detailed description for the different modalities.

We present a method to measure intrapersonal synchronization of movement from motion capture data, and we show that our method is effective in classifying the level of skills of athletes performing karate kata. Our method is based on detecting relevant peaks of acceleration of limbs (arms and legs) and measuring their synchronization. We run a multiscale analysis, based on topological persistence, to rank the importance of peaks of acceleration. The resulting impulse signals are processed next with a multievent class synchronization algorithm, in order to define an overall synchronization index that scores the level of intrapersonal synchronization with a single scalar value. We build a basic multiclass classifier, which uses just the means of indexes computed on the different classes in the training set. We make a statistical analysis and a cross validation of the classifier on real data. Performances by athletes from three levels of skill have been recorded, classified by experts, and used to test our method. Cross validation of the classifier is performed by leave-one-out and bootstrap resampling. Results show that our method can classify correctly with very high probability (beyond 99%), while it succeeds on 100% of the data used in cross validation.

Full-body human movement is characterized by fine-grain expressive qualities that humans are easily capable of exhibiting and recognizing in others’ movement. In sports (e.g., martial arts) and performing arts (e.g., dance), the same sequence of movements can be performed in a wide range of ways characterized by different qualities, often in terms of subtle (spatial and temporal) perturbations of the movement. Even a non-expert observer can distinguish between a top-level and average performance by a dancer or martial artist. The difference is not in the performed movements--the same in both cases--but in the “quality” of their performance. In this article, we present a computational framework aimed at an automated approximate measure of movement quality in full-body physical activities. Starting from motion capture data, the framework computes low-level (e.g., a limb velocity) and high-level (e.g., synchronization between different limbs) movement features. Then, this vector of features is integrated to compute a value aimed at providing a quantitative assessment of movement quality approximating the evaluation that an external expert observer would give of the same sequence of movements. Next, a system representing a concrete implementation of the framework is proposed. Karate is adopted as a testbed. We selected two different katas (i.e., detailed choreographies of movements in karate) characterized by different overall attitudes and expressions (aggressiveness, meditation), and we asked seven athletes, having various levels of experience and age, to perform them. Motion capture data were collected from the performances and were analyzed with the system. The results of the automated analysis were compared with the scores given by 14 karate experts who rated the same performances. Results show that the movement-quality scores computed by the system and the ratings given by the human observers are highly correlated (Pearson’s correlations r = 0.84, p = 0.001 and r = 0.75, p = 0.005).

Analysis of human movement data is a core topic of many research studies in human-human and human-computer interaction. Whereas, on the one side, automated movement analysis is often based on the application of sophisticated computer science techniques (e.g., motion tracking from video recordings), on the other side the interdisciplinary nature of research in this area requires the availability of tools that can be used by researchers who may not have an advanced computer science expertise. This paper presents a system enabling users, who are not necessarily computer scientists, to perform motion tracking from a dataset of video recordings. The system - consisting of a set of (freely downloadable) tools accessible by means of user friendly graphical interfaces - was designed, developed, and tested in the context of a project for automated analysis of entrainment in ensemble music performance, following the needs and requirements of musicologists and psychologists.

In this paper we present a low-intrusive approach to the detection of expressive full-body movement qualities. We focus on two qualities: Lightness and Fragility and we detect them using the data captured by four wearable devices, two Inertial Movement Units (IMU) and two electromyographs (EMG), placed on the forearms. The work we present in the paper stems from a strict collaboration with expressive movement experts (e.g., contemporary dance choreographers) for defining a vocabulary of basic movement qualities. We recorded 13 dancers performing movements expressing the qualities under investigation. The recordings were next segmented and the perceived level of each quality for each segment was ranked by 5 experts using a 5-points Likert scale. We obtained a dataset of 150 segments of movement expressing Fragility and/or Lightness. In the second part of the paper, we define a set of features on IMU and EMG data and we extract them on the recorded corpus. We finally applied a set of supervised machine learning techniques to classify the segments. The best results for the whole dataset were obtained with a Naive Bayes classifier for Lightness (F-score 0.77), and with a Support Vector Machine classifier for Fragility (F-score 0.77). Our approach can be used in ecological contexts e.g., during artistic performances.

Laughter is a social signal capable of facilitating interaction in groups of people: it communicates interest, helps to improve creativity, and facilitates sociability. This article focuses on: endowing virtual characters with computational models of laughter synthesis, based on an expressivity-copying paradigm; evaluating how the physically co-presence of the laughing character impacts on the user’s perception of an audio stimulus and mood. We adopt music as a means to stimulate laughter. Results show that the character presence influences the user’s perception of music and mood. Expressivity-copying has an influence on the user’s perception of music, but does not have any significant impact on mood.

Intrapersonal synchronization of limb movements is a relevant feature for assessing coordination of motoric behavior. In this paper, we show that it can also distinguish between full-body movements performed with different expressive qualities, namely rigidity, fluidity, and impulsivity. For this purpose, we collected a dataset of movements performed by professional dancers, and annotated the perceived movement qualities with the help of a group of experts in expressive movement analysis. We computed intra personal synchronization by applying the Event Synchronization algorithm to the time-series of the speed of arms and hands. Results show that movements performed with different qualities display a significantly different amount of intra personal synchronization: impulsive movements are the most synchronized, the fluid ones show the lowest values of synchronization, and the rigid ones lay in between.

Chronic (persistent) pain (CP) affects 1 in 10 adults; clinical resources are insufficient, and anxiety about activity restricts lives. Technological aids monitor activity but lack necessary psychological support. This article proposes a new sonification framework, Go-with-the-Flow, informed by physiotherapists and people with CP. The framework proposes articulation of user-defined sonified exercise spaces (SESs) tailored to psychological needs and physical capabilities that enhance body and movement awareness to rebuild confidence in physical activity. A smartphone-based wearable device and a Kinect-based device were designed based on the framework to track movement and breathing and sonify them during physical activity. In control studies conducted to evaluate the sonification strategies, people with CP reported increased performance, motivation, awareness of movement, and relaxation with sound feedback. Home studies, a focus group, and a survey of CP patients conducted at the end of a hospital pain management session provided an in-depth understanding of how different aspects of the SESs and their calibration can facilitate self-directed rehabilitation and how the wearable version of the device can facilitate transfer of gains from exercise to feared or demanding activities in real life. We conclude by discussing the implications of our findings on the design of technology for physical rehabilitation.

In this paper, we investigate the detection of laughter from the user's nonverbal full-body movement in social and ecological contexts. Eight hundred and one laughter and nonlaughter segments of full-body movement were examined from a corpus of motion capture data of subjects participating in social activities that stimulated laughter. A set of 13 full-body movement features was identified, and corresponding automated extraction algorithms were developed. These features were extracted from the laughter and nonlaughter segments, and the resulting dataset was provided as input to supervised machine learning techniques. Both discriminative (radial basis function-support vector machines, k-nearest neighbor, and random forest) and probabilistic (naive Bayes and logistic regression) classifiers were trained and evaluated. A comparison of automated classification with the ratings of human observers for the same laughter and nonlaughter segments showed that the performance of our approach for automated laughter detection is comparable with that of humans. The highest F-score (0.74) was obtained by the random forest classifier, whereas the F-score obtained by human observers was 0.70. Based on the analysis techniques introduced in the paper, a vision-based system prototype for automated laughter detection was designed and evaluated. Support vector machines (SVMs) and Kohonen's self-organizing maps were used for training, and the highest F-score was obtained with SVM (0.73).

We present a computational model and a system for the automated recognition of emotions starting from full-body movement. Three-dimensional motion data of full-body movements are obtained either from professional optical motion-capture systems (Qualisys) or from low-cost RGB-D sensors (Kinect and Kinect2). A number of features are then automatically extracted at different levels, from kinematics of a single joint to more global expressive features inspired by psychology and humanistic theories (e.g., contraction index, fluidity, and impulsiveness). An abstraction layer based on dictionary learning further processes these movement features to increase the model generality and to deal with intraclass variability, noise, and incomplete information characterizing emotion expression in human movement. The resulting feature vector is the input for a classifier performing real-time automatic emotion recognition based on linear support vector machines. The recognition performance of the proposed model is presented and discussed, including the tradeoff between precision of the tracking measures (we compare the Kinect RGB-D sensor and the Qualisys motion-capture system) versus dimension of the training dataset. The resulting model and system have been successfully applied in the development of serious games for helping autistic children learn to recognize and express emotions by means of their full-body movement.

In our demo, LoL, a user interacts with a virtual agentable to copy and to adapt its laughing and expressive behaviorson-the-fly. Our aim is to study copying capabilitiesparticipate in enhancing user’s experience in the interaction.User listens to funny audio stimuli in the presenceof a laughing agent: when funniness of audio increases, theagent laughs and the quality of its body movement (directionand amplitude of laughter movements) is modulated on-theflyby user’s body features.

How efficiently decoding affective information when computational resources and sensor systems are limited? This paper presents a framework for analysis of affective behavior starting with a reduced amount of visual information related to human upper-body movements. The main goal is to individuate a minimal representation of emotional displays based on non-verbal gesture features. The GEMEP (Geneva multimodal emotion portrayals) corpus was used to validate this framework. Twelve emotions expressed by ten actors form the selected data set of emotion portrayals. Visual tracking of trajectories of head and hands was performed from a frontal and a lateral view. Postural/shape and dynamic expressive gesture features were identified and analyzed. A feature reduction procedure was carried out, resulting in a four-dimensional model of emotion expression, that effectively classified/grouped emotions according to their valence (positive, negative) and arousal (high, low). These results show that emotionally relevant information can be detected/measured/obtained from the dynamic qualities of gesture. The framework was implemented as software modules (plug-ins) extending the EyesWeb XMI Expressive Gesture Processing Library and was tested as a component for a multimodal search engine in collaboration with Google within the EU-ICT I-SEARCH project.

In this paper, we study perception of intensity in-congruence between auditory and visual modalities of synthesized expressions of laughter. In particular, we investigate whether incongruent expressions are perceived as 1) regulated, and 2) unsuccessful in terms of animation synthesis. For this purpose, we conducted a perceptive study with the use of a virtual agent. Congruent and incongruent multimodal expressions of laughter were synthesized from natural audiovisual laughter episodes, using machine learning algorithms. Next, the intensity of facial expressions and body movements were systematically manipulated to check whether the resulting incongruent expressions are perceived differently compared to the corresponding congruent expressions. Results show that 1) intensity incongruence lowers the perception of believability and plausibility, and 2) the in-congruent laughter expressions displaying high intensity in the audio modality and low intensity in the body movement and facial expression are perceived as more fake than the corresponding congruent expressions. Such results have implications for both animation synthesis as well as expression regulation research.

This is a short description of the Emotional Charades serious game demo. Our goal is to focus on emotion expression through body gestures, making the players aware of the amount of affective information their bodies convey. The whole framework aims at helping children with autism to understand and express emotions. We also want to compare the performances of our automatic recognition system and the ones achieved by humans.

The present study investigates expressive non-verbal interaction in musical context starting from behavioral features extracted at individual and group level. We define four features related to head movement and direction that may help gaining insight on the expressivity and cohesion of the performance. Our preliminary findings obtained from the analysis of a string quartet recorded in ecological settings show that these features may help in distinguishing between two types of performance: (a) a concert-like condition where all musicians aim at performing at best, (b) a perturbed one where the 1st violinist devises alternative interpretations of the music score without discussing them with the other musicians.

When people perform a task as part of a joint action, their behavior is not the same as it would be if they were performing the same task alone: it is adapted to facilitate shared understanding (or sometimes to prevent it). Joint performance of music offers a test bed for ecologically valid investigations of the way non-verbal behavior facilitates joint action. Here we compare the expressive of violinists when playing solo Vs. in the string quartet music ensemble. The first and second violinists of a famous concert string quartet were asked to play the same musical fragment in a solo condition and with the quartet. Synchronized multimodal recordings have been created from the performances, using a specially developed software platform. The differences are not obvious to untrained observers but they are discriminated by musicians, and appropriate measures show that they exist. In particular, using an appropriate measure of entropy shows that head movements are more predictable in the quartet scenario. The change does not, as might be assumed, entail markedly reduced expression. The data pose provocative questions about joint action in realistically complex scenarios.

Within the EU ILHAIRE Project, researchers of several disciplines (e.g., computer sciences, psychology) collaborate to investigate the psychological foundations of laughter, and to bring this knowledge into shape for the use in new technologies (i.e., affective computing). Within this framework, in order to endow machines with laughter capabilities (encoding as well as decoding), one crucial task is an adequate description of laughter in terms of morphology. In this paper we present a work methodology towards automated full body laughter detection: starting from expert annotations of laughter videos we aim to identify the body features that characterize laughter.

The Art Exhibition of ACM Multimedia 2013 has attracted significant work from a variety of digital artists collaborating with research institutions. We have endeavored to select exhibits that achieved an interesting balance between technology and artistic intent. The techniques underpinning these artworks are relevant to several technical tracks of the conference, in particular those dealing with human-centered and interactive media. We briefly review how the various installations revisit current topics in Multimedia research, focusing more specifically on their approach to dynamic content generation, user experience, multimodality, and affective interfaces. Once again, the unique blend of technology and user experience does not limit itself to showcasing recent advances in interactive media, and should be of interest to all conference participants as well as the general public the exhibition space will be open to.

Since its first edition in 2009, the International Workshop on Social Behaviour in Music (SBM) has been an occasion for researchers and practitioners for discussing recent advances in automated analysis of social behaviour, being music the selected test-bed and application scenario. The first edition of SBM was held in Vancouver, Canada, in the framework of the 2009 IEEE International Conference on Social Computing (SocialCom 2009). The second one was held in Genova, Italy, in the framework of the 4th International ICST Conference on Intelligent Technologies for Interactive Entertainment (Intetain 2011). SBM is now at its third edition, which takes place in the framework of the 14th International Conference on Multimodal Interaction (ICMI 2012), Santa Monica, California, USA. Again, SBM aims at providing a picture of current research breakthrough and issues, giving at the same time directions for future works and collaborations.

In this paper, we report on work around the I-SEARCH EU (FP7 ICT STREP) project whose objective is the development of a multimodal search engine targeted at mobile and desktop devices. Each of these device classes has its specific hardware capabilities and set of supported features. In order to provide a common multimodal search experience across device classes, one size does not fit all. We highlight ways to achieve the same functionality agnostic of the device class being used for the search, and present concrete use cases.

This paper presents a framework for analysis of affective behavior starting with a reduced amount of visual information related to human upper-body movements. The main goal is to individuate a minimal representation of emotional displays based on nonverbal gesture features. The GEMEP (Geneva multimodal emotion portrayals) corpus was used to validate this framework. Twelve emotions expressed by 10 actors form the selected data set of emotion portrayals. Visual tracking of trajectories of head and hands were performed from a frontal and a lateral view. Postural/shape and dynamic expressive gesture features were identified and analyzed. A feature reduction procedure was carried out, resulting in a 4D model of emotion expression that effectively classified/grouped emotions according to their valence (positive, negative) and arousal (high, low). These results show that emotionally relevant information can be detected/measured/obtained from the dynamic qualities of gesture. The framework was implemented as software modules (plug-ins) extending the EyesWeb XMI Expressive Gesture Processing Library and is going to be used in user centric, networked media applications, including future mobiles, characterized by low computational resources, and limited sensor systems.

Our research focused on ensemble musical performance, an ideal test-bed for the development of models and techniques for measuring creative social interaction in an ecologically valid framework. Starting from expressive behavioral data of a string quartet, this paper addresses the application of Multi-Scale Entropy method to investigate dominance.

This paper presents a multimodal system for real-time analysis of nonverbal affective social interaction in small groups of users. The focus is on two major aspects of affective social interaction: the synchronization of the affective behavior within a small group and the emergence of functional roles, such as leadership. A small group of users is modeled as a complex system consisting of single interacting components that can auto-organize and show global properties. Techniques are developed for computing quantitative measures of both synchronization and leadership. Music is selected as experimental test-bed since it is a clear example of interactive and social activity, where affective nonverbal communication plays a fundamental role. The system has been implemented as software modules for the EyesWeb XMI platform (http://www.eyesweb.org). It has been used in experimental frameworks (a violin duo and a string quartet) and in real-world applications (in user-centric applications for active music listening). Further application scenarios include entertainment, edutainment, therapy and rehabilitation, cultural heritage, and museum applications. Research has been carried out in the framework of the EU-ICT FP7 Project SAME (http://www.sameproject.eu).

This work aims at defining a computational model of human emotional entrainment. Music, as a non-verbal language to express emotions, is chosen as an ideal test bed for these aims. We start from multimodal gesture and motion signals, recorded in a real world collaborative condition in an ecological setting. Four violin players were asked to play, alone or in duo, a music fragment in two different perceptual feedback modalities and in four different emotional states. We focused our attention on phase synchronisation of the head motions of the players. From observation by subjects (musicians and observers), an evidence of entrainment emerges between players. The preliminary results, based on a reduced data set, however do not grasp fully this phenomenon. A more extended analysis is current subject of investigation.

This paper focuses on multimodal and cross-modal analysis of expressive gesture with a particular focus on collaborative interactive systems exploiting tangible acoustic interfaces (TAIs). We developed TAIs aiming at processing expressive information from users and supporting creativity in concrete music theatre and museum projects. The paper presents (i) techniques for extraction and analysis of high-level features from expressive gesture of TAIs users in collaborative frameworks, (ii) concrete examples of multimodal and cross-modal processing of expressive gesture, (iii) examples of how such results have been exploited in public events and artistic productions. In such occasions the developed techniques have been applied and evaluated with experiments involving both experts and the general audience. Research is carried out in the framework of the EU-IST STREP Project TAI-CHI (Tangible Acoustic Interfaces for Computer-Human Interaction). High-level expressive gesture analysis and multimodal and cross-modal processing are achieved in the new EyesWeb 4 open platform (available at www.eyesweb.org)

Abstract This paper presents and discusses some experiments having the purpose of planning, developing and validating aesthetically resonant environments for different types of sensorimotor impairments which affect Parkinson's patients. From a technical point of view the aim is to develop a computational open architecture in which it is possible to integrate modules for gesture analysis and recognition and for interactive construction of therapeutic exercises based on multimedia stimulation in real time. The clinical objective is to experiment with a device of sensorimotor stimulation that supports akinesia compensation by controlling movement rhythmic structures in subjects with Parkinson's disease. The EyesWeb open architecture has been used to analyse patients' motion and to produce visual feedback during therapy sessions in real time. A pilot study has been conducted on two Parkinson's disease patients in the framework of the EU IST project CARE‐HERE. Copyright © 2003 John Wiley & Sons, Ltd.

We introduce the model of artificial emotions embedded in our architecture of emotional agents. An emotional agent also possesses rational knowledge and reactive capabilities, and is able to interact with the external world, including other agents. Emotional agents can give rise to more effective, natural, and stimulating interaction with humans. Our model and related agent architecture has been developed and applied in several multimedia-multimodal application environments.

We present a CASE tool based on high level Petri nets (PNs) for the design of real-time simulators of plants, it is derived from XpetreX whose peculiar characteristics have been improved towards a more effective object oriented approach. The CASE tool basically consists of two main software modules: a development system and a run-time system. The former supports the user in the design process of the simulator. The latter is the real-time platform able to run the model generated by the development system. The object oriented approach allows the user to design reusable modules for the automatic generation of PNs by subnet class derivation and instantiation. A main goal is to use PNs for industrial plants simulation even to those users that have no or little knowledge on PNs. A library of classes can be written in the same way as a library of object in an object oriented programming language. The ability of the run-time module to communicate with other processes is being implemented and allows the interaction with other applications.

The purpose of our work is to realize an agent for intelligent interaction between humans and robots, embedding a computational model of artificial emotions with learning and self-adaptation features. The gesture of a human can cause the robot to change its "emotional state", which is exhibited by means of integrated visual media, environmental lights, music, and changes in robot's style of movement and behavior. We adopt our emotional agent architecture for developing the agent. The system is a kind of "emotional activator", which plays the role of stimulating the human creativity. In our environment, humans do not need to wear any special on-body sensor. We use the small robot on wheels, equipped with on-board ultrasound sensors. Furthermore, We develop a camera-based sensor system to increase its input capabilities.

Strong deficiencies are present in symbolic models for action representation and planning, regarding mainly the difficulty of coping with real, complex environments. These deficiencies can be attributed to several problems, such as the inadequacy in coping with incompletely structured situations, the difficulty of interacting with visual and motorial aspects, the difficulty in representing low-level knowledge, the need to specify the problem at a high level of detail, and so on. Besides the purely symbolic approaches, several nonsymbolic models have been developed, such as the recent class of subsym-bolic techniques. A promising paradigm for the modeling of reasoning, which combines features of both symbolic and analogical approaches, is based on the construction of analogical models of the reference for the internal representations, as introduced by Johnson-Laird. In this work, we propose a similar approach to the problem of knowledge representation and reasoning about actions and plans. We propose a hybrid approach, symbolic and analogical, in which the inferences are partially devolved to measurements on analogical models generated starting from the symbolic representation. the interaction between the symbolic and the analogical level is due to the fact that procedures are connected to some symbols, allowing generating, updating, and verifying the mental model. the hybrid model utilizes, for the symbolic component, a representation system based on the distinction between terminological and assertional knowledge. the terminological component adopts a SI-Net formalism, extended by temporal primitives. the assertional component is a subset of first-order logics. the analogical representation is a set of concurrent procedures modeling parts of the world, action processes, simulations, and metaphors based on force fields concepts. A particular case study, regarding the problem of the assembly of a complex object from parts, is taken as an experimental paradigm. © 1993 John Wiley Sons, Inc.

A presentation is made of the methodology adopted in the design of the coordination subsystem and the real-time scheduling subsystem for the simulation and the real-time control of flexible manufacturing systems (FMSs). This particular approach, combining structured, object-oriented paradigms and a particular extension of high-level Petri nets, is presented and discussed. The focus is on the following issues: (1) the modeling of the FMS control system in a structured knowledge base (frame-based multiple-inheritance semantic networks); (2) the automatic synthesis of Petri net models of the coordination subsystem, starting from such a knowledge base; (3) the dynamic behavior of the coordination subsystem; and (4) the structure of the real-time scheduling subsystem, based on particular heuristic suboptimal multicriterion algorithms. The interaction mechanisms between the coordination and scheduling subsystems are defined. A particular case study involving a class of problems on the concurrent scheduling of multiple projects is taken as an example.>