Non-lexical sounds such as laughter are considered important discourse markers in conversation as these sounds complement the lexical information in shaping interpersonal relations, managing the conversation, and in expressing attitudes and affect. However, general purpose open-source automatic speech recognition (ASR) systems are typically not focusing on these sounds. Laughter transcription is an understudied task in ASR: laughter is often not modelled as a token and is discarded in the evaluation of ASR. In our study, we investigate how current open-source ASR models are handling laughter and speech-laugh (speech interspersed with laughter). Using Switchboard and BuckEye as conversational speech corpora, we fine-tuned and evaluated Whisper and Wav2Vec2. Our results show that laughter can be integrated in ASR transcriptions without substantially degrading word error rate.

Understanding how users engage with socially interactive agents in immersive cultural environments is crucial for designing adaptive and contextually appropriate virtual experiences.This study explores how individual user traits-including familiarity with virtual reality, museum visitation habits, prior knowledge, interest in specific artworks, and interaction preferences-relate to gaze behavior and conversational engagement in a VR art exhibition.Fifty-two participants explored a virtual museum featuring five paintings and interacted with agents designed to share information and facilitate discussion.Data were collected through questionnaires, eye-tracking metrics (e.g., fixation duration, scanpath length), and conversational transcripts (e.g., response length, number of user turns).A multimodal analysis aligned gaze and speech data to examine how visual attention and verbal interactions co-occur in relation to user traits.Results indicate that traits such as prior knowledge, interest, and personalization preferences significantly influence both gaze patterns and conversational behaviors.While some effects varied in strength or consistency, the findings offer valuable insights into how personal characteristics shape user-agent interactions.These insights inform the design of adaptive virtual agents and support broader efforts to model multimodal user bahevior for real-time personalization in immersive environment.

Accurate interpretation of the environment is both essential to automated robots and highly beneficial for teleoperated robots. Going beyond obstacle recognition, interpreting the semantics of the environment and the actions it affords, enables robots to interact with environments made for humans in a human-like manner. This paper describes the collection of affordance labels at the object level for multiple indoors datasets, to train computer vision algorithms for detecting object affordances in indoor spaces. It is a first step towards determining high level "semantic" affordances, to allow reasoning about what to do with objects, rather than "functional" affordances, which allow reasoning about how to use the same. A baseline model is provided, which highlights the value of the affordance labels in a variety of robotics applications.

The human eye plays a crucial role in shaping social judgments; for instance, maintaining eye contact enhances social interactions and influences how others perceive us. At a more subtle level, the pupils also play an important role; recent research shows that pupil mimicry fosters trust. In our digitizing world, people frequently interact with virtual agents. Therefore, it is of crucial importance to investigate whether the positive effects of eye gaze and pupil mimicry generalize to virtual interactions. In the current study, we investigated whether eye contact and pupil dilation mimicry influence the evaluation of virtual humans and whether these effects vary with the participants’ levels of social anxiety. Six virtual humans told a story which participants watched on a monitor screen. Three of them were programmed to mimic the participants’ pupil size at moments when participants displayed natural pupil dilation, while the other three did not mimic these instances and instead, their pupil sizes dilated at random moments. After each story, we assessed how the virtual human was perceived by taking multiple social evaluation measures. The results show that the more the participants made eye contact, the more positively they evaluated the virtual human, especially during trials where the virtual humans mimicked the spontaneous instances of pupil dilation in participants’ eyes. Social anxiety level did not influence the results. This study demonstrates that the cumulative positive effects of eye contact and pupil dilation mimicry generalize to interactions with virtual agents, which may inspire potential applications.

In this article, we transition from the theoretical and experimental groundwork of manipulating and measuring Sense of Embodiment (SoE) to addressing a fundamental question: What is the purpose of optimizing the SoE in a teleoperation system? This exploration centers on investigating the potential positive effects of SoE on motor adaptation, the acceleration of motor learning, and the potential enhancement of task performance. The article delves into this investigation by focusing on two critical research questions: (1) what is the effect of SoE on task performance in a perceptual-motor task? (2) What is the effect of SoE on the asymptote of the learning curve in a perceptual-motor task? Drawing insights from the existing literature, the hypothesis emerges that enhancing SoE yields positive effects not only on task performance (H1) but also on the overall embodiment experience (H2). An additional layer of exploration is introduced through an exploratory research question: Are these results consistent across diverse scenarios and tasks? The study design encompasses two distinct user studies, each set in different applications and featuring various avatars, yet all anchored in similar tasks, specifically a modified peg-in-hole task: (1) in the first experiment, participants operated a robotic arm with a human-like hand as end-effector, and they were required to perform a classic peg-in-hole task; (2) in user study 2, the task is transformed into a variation that we called “peg-on-button,” wherein participants use a robotic arm with a gripper as the end-effector to press a lit button. In both studies, a consistent pattern emerges: a setup that fosters embodiment has a positive impact on motor learning and adaptation, resulting in improved task performance. A supportive setup also reduces the perception of the surrogate as a mere mediator between the operator and the remote environment, especially when contrasted with a setup that suppresses embodiment. The positive effects on motor learning and task performance advocate for the incorporation of embodiment-supportive designs in teleoperational setups. However, the nuanced relationship between SoE and long-term task performance prompts a call for further exploration and consideration of various factors influencing teleoperation outcomes across diverse scenarios and tasks.

Technology allows runners to obtain information on their running parameters and gain insights to self-reflect on their performance and technique. One of the higher levels of self reflection is transformative self-reflection, which engages runners in a thought process to adjust future efforts. Observing one-self in a video as well as visualizing running parameters have shown benefits in supporting self-reflection; combining the two thus presents an interesting research opportunity. This paper presents the process of designing a dashboard where abstract visuals are combined with drone video to promote runners’ self-reflection. A user centered approach was employed to select the parameters, design the visuals, and evaluate the dashboard. Evaluation using TSRI substantiated the dashboard’s capabilities to provide runners valuable insights and support their running data exploration. Interviews highlighted the dashboard’s potential in promoting self-reflection and provided design considerations for future iterations.

In daily conversations, people often encounter problems prompting conversational repair to enhance mutual understanding.By employing an automatic coreference solver, alongside examining repetition, we identify various linguistic features that distinguish turns when the addressee initiates repair from those when they do not.Our findings reveal distinct patterns that characterize the repair sequence and each type of other-repair initiation.

During social interactions, people often unintentionally mimic each other's behaviors, which can occur at the motor level (e.g. smiles or head movements), and the autonomic level (e.g. pupil size). Mimicry has often been linked with increases in liking and trust and tends to smoothen the bond between interaction partners. However, people with social anxiety are generally less expressive and they may also have altered mimicry behavior. The current study aims to explore the relationship between mimicry, person-perception, and social anxiety levels by having participants interact with virtual humans. Compared to real interaction partners or photographed or filmed actors, virtual humans can be controlled and behaviors very precisely manipulated. Participants observed ten virtual humans telling a neutral story on a computer monitor. At predetermined time points, the virtual humans showed four types of expressions: smiles, head movements, pupil dilation, and blushing. After each story, we measured to what extent participants liked and trusted the virtual humans via the avatar evaluation questionnaire, the desire for future interaction scale, and an investment game. In addition, we assessed participants’ social anxiety levels. The results show that participants, regardless of anxiety levels, mimicked the virtual humans’ smiles and that smile mimicry is associated with an increased liking and trust toward the virtual humans. These results build on previous research showing that the mimicry of smiles acts as a social glue and also benefits virtual social interactions.

As the demand for socially intelligent Virtual Humans (VHs) increases, so follows the demand for effective and efficient cross-discipline collaboration that is required to bring these VHs “to life”. One avenue for increasing cross-discipline fluency is the aggregation and organization of seemingly disparate areas of research and development (e.g., graphics and emotion models) that are essential to the field of VH research. Our initial investigation (1) identifies and catalogues research streams concentrated in three multidisciplinary VH topic clusters within the domain of social intelligence, Emotion, Social Behavior, and The Face, (2) brings to the forefront key themes and prolific authors within each topic cluster, and (3) provides evidence that a full scoping review is warranted to further map the field, aggregate research findings, and identify gaps in the research. To enable collaboration, we provide full access to the refined VH cluster datasets, key word and author word clouds, as well as interactive evidence maps.

There is an underexplored interaction space for drones that can be utilised as running interaction technology, distinct from human drone interaction that warrants foregrounding. This paper consolidates the current state of art in running interaction technology through a review of relevant studies and commercial technologies in a framework positioned using dimensions related to the form of interaction as identified in the sports ITECH framework. Our analysis highlights the unmet opportunities in running interaction technology and presents the potential of drones to further support runners. The potential of drones to support various forms of interaction are supported using exemplar research done in human-drone interaction. Through our findings, we hope to inform and expedite future research and practice in the field of running interaction technology and runner drone interaction by supporting researchers in defining and situating their contributions.

We present a literature review and a toolbox to help the reader find the best method to design for and assess Sense of Embodiment (SoE) in several application scenarios. The main examples are based on teleoperation applications, due the challenges that these applications present. The three embodiment components that we consider to describe SoE are sense of ownership, sense of agency, and sense of self-location. We relate each embodiment component to the most often used assessment measures, test tasks, and application scenarios. The toolbox is built to efficiently design, test, and assess an embodiment experience, following seven concrete steps. We provide four main contributions: 1) a literature review of the assessment measures and strategies used to measure SoE; 2) a systematic categorization of SoE measures; 3) A categorization of the main test tasks used in SoE assessment; and 4) a toolbox consisting of seven steps as guidance to design SoE. We included several examples and tables to guide the user step by step through the design of an embodiment experience.

Given the importance of affective touch in human interactions, technology designers are increasingly attempting to bring this modality to the core of interactive technology. Advances in haptics and touch-sensing technology have been critical to fostering interest in this area. In this survey, we review how affective touch is investigated to enhance and support the human experience with or through technology. We explore this question across three different research areas to highlight their epistemology, main findings, and the challenges that persist. First, we review affective touch technology through the human–computer interaction literature to understand how it has been applied to the mediation of human–human interaction and its roles in other human interactions particularly with oneself, augmented objects/media, and affect-aware devices. We further highlight the datasets and methods that have been investigated for automatic detection and interpretation of affective touch in this area. In addition, we discuss the modalities of affective touch expressions in both humans and technology in these interactions. Second, we separately review how affective touch has been explored in human–robot and real-human–virtual-human interactions where the technical challenges encountered and the types of experience aimed at are different. We conclude with a discussion of the gaps and challenges that emerge from the review to steer research in directions that are critical for advancing affective touch technology and recognition systems. In our discussion, we also raise ethical issues that should be considered for responsible innovation in this growing area.

Laughter is an affective and social signal that serves many functions. Similar to other social affective signals, laughter production and perception is at least partially context dependent. Familiarity of conversation partners has been shown to be a contextual influence on laughter production. However, the literature is still scarce and divided on how this complex interaction between familiarity and laughter works. Our goal with this paper is to further study this interaction using a newly acquired and annotated corpus and contrast our findings with existing findings in a comprehensive overview of the literature. Using a series of Linear Mixed-Effect Models, we studied if familiarity with the conversational partner and the sex of same-sex conversation pairs affect the laughter frequency, co-laughter frequency or laughter acoustics produced by the subjects in the corpus. The model outputs show that the frequency of laughter and co-laughter is not influenced by familiarity or the sex of conversation pairs. Interestingly, the percentage of co-laughs is significantly influenced by familiarity. Laughter voicedness is influenced by both the familiarity and sex of conversation pairs, where duration of laughter is only influenced by familiarity of the conversation pair. We conclude that familiarity of conversation pairs play an important role in laughter production during interactions and should be systematically explored. Furthermore we make several suggestions for improving the methods in future work.

The development of applications with intelligent virtual agents (IVA) often comes with integration of multiple complex components. In this article we present the Agents United Platform: an open source platform that researchers and developers can use as a starting point to setup their own multi-IVA applications.

Unpredictability in robot behaviour can cause difficulties in interacting with robots. However, for social interactions with robots, a degree of unpredictability in robot behaviour may be desirable for facilitating engagement and increasing the attribution of mental states to the robot. To generate a better conceptual understanding of predictability, we looked at two facets of predictability, namely, the ability to predict robot actions and the association of predictability as an attribute of the robot. We carried out a video human-robot interaction study where we manipulated whether participants could either see the cause of a robot’s responsive action or could not see this, because there was no cause, or because we obstructed the visual cues. Our results indicate that when the cause of the robot’s responsive actions was not visible, participants rated the robot as more unpredictable and less competent, compared to when it was visible. The relationship between seeing the cause of the responsive actions and the attribution of competence was partially mediated by the attribution of unpredictability to the robot. We argue that the effects of unpredictability may be mitigated when the robot identifies when a person may not be aware of what the robot wants to respond to and uses additional actions to make its response predictable.

Predictability is important to autistic individuals, and robots have been suggested to meet this need as they can be programmed to be predictable, as well as elicit social interaction. The effectiveness of robot-assisted interventions designed for social skill learning presumably depends on the interplay between robot predictability, engagement in learning, and the individual differences between different autistic children. To better understand this interplay, we report on a study where 24 autistic children participated in a robot-assisted intervention. We manipulated the variance in the robot’s behaviour as a way to vary predictability, and measured the children’s behavioural engagement, visual attention, as well as their individual factors. We found that the children will continue engaging in the activity behaviourally, but may start to pay less visual attention over time to activity-relevant locations when the robot is less predictable. Instead, they increasingly start to look away from the activity. Ultimately, this could negatively influence learning, in particular for tasks with a visual component. Furthermore, severity of autistic features and expressive language ability had a significant impact on behavioural engagement. We consider our results as preliminary evidence that robot predictability is an important factor for keeping children in a state where learning can occur.

Eating is in many ways a social activity. Yet, little is known about the social dimension of eating influencing individual eating habits. Nor do we know much about how to purposefully design for interactions in the social space of eating. This paper presents (1) the journey of exploring the social space of eating by designing an artifact, and (2) the actual artifact designed for the purpose of exploring the interaction dynamics of social eating. The result of this Research through Design journey is the Sensory Interactive Table: an interactive dining table based on explorations of the social space of eating, and a probe to explore the social space of eating further.

Although laughter has gained considerable interest from a diversity of research areas, there still is a need for laughter specific databases. We present the Multimodal Laughter during Interaction (MULAI) database to study the expressive patterns of conversational and humour related laughter. The MULAI database contains 2 hours and 14 minutes of recorded and annotated dyadic human-human interactions and includes 601 laughs, 168 speech-laughs and 538 on- or offset respirations. This database is unique in several ways; 1) it focuses on different types of social laughter including conversational- and humour related laughter, 2) it contains annotations from participants, who understand the social context, on how humourous they perceived themselves and their interlocutor during each task, and 3) it contains data rarely captured by other laughter databases including participant personality profiles and physiological responses. We use the MULAI database to explore the link between acoustic laughter properties and annotated humour ratings over two settings. The results reveal that the duration, pitch and intensity of laughs from participants do not correlate with their own perception of how humourous they are, however the acoustics of laughter do correlate with how humourous they are being perceived by their conversational partner.

An important aspect of human emotion perception is the use of contextual information to understand others' feelings even in situations where their behavior is not very expressive or has an emotionally ambiguous meaning. For technology to successfully detect affect, it must mimic this human ability when analyzing audiovisual input. Databases upon which machine learning algorithms are trained should capture the context of social interactions as well as the behavior expressed in them. However, there is a lack of consensus about what constitutes relevant context in such databases. In this article, we make two contributions towards overcoming this challenge: (a) we identify two principal sources of context for emotion perceptions based on psychological theory, and (b) we provide an overview of how each of these has been considered in published databases covering social interactions. Our results show that a similar set of contextual features are present across the reviewed databases. Between all the different databases researchers seem to have taken into account a set of contextual features reflecting the sources of context seen in psychological theory. However, within individual databases, these features are not yet systematically varied. This is problematic because it prevents them from being used directly as resources for the modeling of context-sensitive affect detection. Based on our findings, we suggest improvements for the future development of affective databases.

In order to contribute to the need of spontaneous multi-modal affective databases for the automatic recognition of emotions in older adults, this paper presents a novel Dutch multi-modal database consisting of emotional memories of older adults. The data consists of positive and negative memories of older adults eliciting through two emotion reliving tasks: autobiographical memory recall in the first session and life story books to discuss these memories in depth in the second session. Data collection was carried out at the participants' home or at a place comfortable to them. Audio was recorded for the first session whereas audio, video and physiological data were recorded for the second session. As this database introduces a novel way of using autobiographical memories to study emotional expressions in older adults, a first step of the complex coding of emotions is presented in this paper. We reflect on the challenges encountered in the database and propose ways to address these issues.

Analyzing emotional valence in spontaneous speech remains complex and challenging. We present an acoustic and lexical analysis of emotional valence in spontaneous speech of older adults. Data was collected by recalling autobiographical memories through a word association task. Due to the complex and personal nature of memories, we propose a novel coding scheme for emotional valence. We explore acoustic properties of speech as well as the use of affective words to predict emotional valence expressed in autobiographical memories. Using mixed-effect regression modelling, we compared predictive models based on acoustic information only, lexical information only, or a combination of both. Results show that the combined model accounts for the highest proportion of explained variance, with the acoustic features accounting for a smaller share of the total variance than the lexical features. Several acoustic and lexical features predicted valence. As a first attempt at analyzing spontaneous emotional speech in older adults autobiographical memories, the study provides more insight in which acoustic features can be used to predict valence (automatically) in a more ecologically valid setting.

This paper presents progress and challenges in developing a platform for multi-character, argumentation based, interaction with a group of virtual coaches for healthcare advice and promotion of healthy behaviours. Several challenges arise in the development of such a platform, e.g., choosing the most effective way of utilising argumentation between the coaches with multiple perspectives, handling the presentation of these perspectives and finally, the personalisation and adaptation of the platform to the user types. In this paper, we present the three main challenges recognized, and show how we aim to address these.

A modern way of life needs a modern way of coaching. Despite the proliferation of ICT solutions for personalized health care, there is still no easy way to provide older adults with integrated coaching services. In this paper we introduce the concept of Council of Coaches — a radically new virtual coaching concept based on multiple autonomous, embodied virtual coaches, which form together a personal council that fulfills the needs of older adults in an integrated way. In this concept, coaching takes the form of an open dialog in which clients co-construct together with a selected number of coaches their own plans to go for a healthier lifestyle. Virtual coaches are presented to users by means of embodied conversational social characters. We discuss technical and social challenges on the path towards realizing the Council of Coaches concept, a radically new view of health coaching that involves the state of the art in human-computer interaction, natural dialogue, and argumentation technology.

We present anew dialogue engine called Flipper 2.0 (Flipper) which aims to help developers of embodied conversational agents (ECAs) to quickly and flexibly create dialogues. Flipper provides a technically stable and robust dialogue management system to integrate with other components of ECAs such as behaviour realisers. We compare Flipper with state-of-the-art dialogue design systems. We describe the details of our dialogue engine, how it handles dialogue management and how it supports the authoring of dialogues. We demonstrate the use of the dialogue engine with examples of design patterns and discuss practical applications. Finally we give recommendations on the cases in which it is beneficial to use Flipper.

Remote mixed reality (MR) collaboration systems allow for multimodal, real-time support from remote experts. We present our open toolkit that provides a flexible end-to-end solution for building such systems using off-the-shelf hardware. From related work, three core design aspects have been identified: 1) the independence of the viewpoint that the visitor (the remote expert) can take in relation to position and viewpoint of the visitee, 2) the immersiveness of the presentation technology for visitor and visitee, and 3) the extent to which the visitor's body is represented in the visitee's environment. This paper describes the implementation of our system, which includes these aspects. In a study aimed at validating whether we implemented these core aspects to good effect, conducted with a collaborative puzzle application built with our toolkit, we examine how variations of these aspects contribute to usability, performance and social presence related metrics.

In our ongoing research, we argue that storytelling activities can be used as creative tasks to stimulate creativity in children, one of the so-called 21st century skills. In this paper, we lay the foundations for our project on digitally supported storytelling, by gathering the viewpoints on storytelling and creativity expressed by local primary school teachers and reviewing methods reported in the literature. We report the main characteristics and features to consider in creative storytelling activities, and present our current work on a distributed storytelling interface for supporting children's creative skills development that meets the practical requirements found in the literature and expressed by the teachers.

Interactive playgrounds are spaces where players engage in collocated, playful activities, in which added digital technology can be designed to promote cognitive, social, and motor skills development. To promote such development, different strategies can be used to implement game mechanics that change player's in-game behavior. One of such strategies is enticing players to take action through incentives akin to game achievements. We explored if this strategy could be used to influence players' proxemic behavior in the Interactive Tag Playground, an installation that enhances the traditional game of tag. We placed the ITP in an art gallery, observed hundreds of play sessions, and refined the mechanics, which consisted in projecting collectible particles around the tagger that upon collection by runners resulted only in the embellishment of their circles. We implemented the refined mechanics in a study with 48 children. The playground automatically collected the players' positions, and analyses show that runners got closer to and moved more towards taggers when using our enticing strategy. This suggests an enticing strategy can be used to influence physical in-game behavior.

Touch displays moving autonomously, i.e., being self-actuated, are starting to be part of system prototypes reported in the literature. In the future, user interactions with such prototypes may take place while the display is moving. However, since current prototypes do not yet require this type of interaction, there is a lack of empirical evidence reporting issues related to touch input in such conditions. This leads us to propose two basic questions: Can we request users to deliver touch gestures during the actuation? Which aspects should we take into account when having touch input in these moving devices? In order to start to answer these questions, we report in this paper a first study to get an insight into how people perform and feel it when they have to carry out touch input on a self-actuated tablet. The preliminary results show that the self-actuated tablet does not necessarily need to be still during touch interaction, and that single-touch gestures based on drag or tap are preferable over others. Furthermore, the results also revealed an issue with tap gestures because of the movement of the tablet.

Research into multimodal flavor perceptions has demonstrated associations between basic tastes and visual cues. Moreover, such associations have been found to influence taste perception. Here we are interested in how mixed reality technology in the form of projection mapping can be used to introduce such visual cues during consumption. First, associations between basic tastes and visualizations that differed in color, shape and animation speed were investigated in a crowdsourcing study. The study demonstrated associations between sweetness and red rounded shapes, and sourness and green angular shapes with a fast animation speed. A subsequent lab study where the visualizations were projected around a cup of yogurt that participants tasted confirmed these associations. Finally, specific combinations of visualizations and animation types were found to influence taste perceptions of the yogurt. The implications of these findings are discussed.

This paper gives a summary of the 2nd International Workshop on Advancements in Social Signal Processing for Multimodal Interaction (ASSP4MI). Following our successful 1st International Workshop on Advancements in Social Signal Processing for Multimodal Interaction, held during ICMI-2015, we proposed the 2nd ASSP4MI workshop during ICMI-2016. The topics addressed and discussions fostered during last year's workshop are considered very relevant and alive in the research community. In this year's workshop, we continued addressing important topics and fostering fruitful discussions among researchers from different disciplines working in the fields of Social Signal Processing (SSP) and multimodal interaction.

We present a demonstration of the ARIA framework, a modular approach for rapid development of virtual humans for information retrieval that have linguistic, emotional, and social skills and a strong personality. We demonstrate the framework's capabilities in a scenario where `Alice in Wonderland', a popular English literature book, is embodied by a virtual human representing Alice. The user can engage in an information exchange dialogue, where Alice acts as the expert on the book, and the user as an interested novice. Besides speech recognition, sophisticated audio-visual behaviour analysis is used to inform the core agent dialogue module about the user's state and intentions, so that it can go beyond simple chat-bot dialogue. The behaviour generation module features a unique new capability of being able to deal gracefully with interruptions of the agent.

This paper describes a substantial effort to build a real-time interactive multimodal dialogue system with a focus on emotional and non-verbal interaction capabilities. The work is motivated by the aim to provide technology with competences in perceiving and producing the emotional and non-verbal behaviours required to sustain a conversational dialogue. We present the Sensitive Artificial Listener (SAL) scenario as a setting which seems particularly suited for the study of emotional and non-verbal behaviour, since it requires only very limited verbal understanding on the part of the machine. This scenario allows us to concentrate on non-verbal capabilities without having to address at the same time the challenges of spoken language understanding, task modeling etc. We first summarise three prototype versions of the SAL scenario, in which the behaviour of the Sensitive Artificial Listener characters was determined by a human operator. These prototypes served the purpose of verifying the effectiveness of the SAL scenario and allowed us to collect data required for building system components for analysing and synthesising the respective behaviours. We then describe the fully autonomous integrated real-time system we created, which combines incremental analysis of user behaviour, dialogue management, and synthesis of speaker and listener behaviour of a SAL character displayed as a virtual agent. We discuss principles that should underlie the evaluation of SAL-type systems. Since the system is designed for modularity and reuse, and since it is publicly available, the SAL system has potential as a joint research tool in the affective computing research community.

Recent research suggests that physical warmth activates perceptions of metaphorical interpersonal warmth and closeness, and increases pro-social behavior. These effects are grounded in our earliest intimate experiences: being held by our loving caregivers. These findings provide reasons to incorporate warmth in devices for distant affective communication, which could simulate one's body heat. An experiment was carried out to gain a better understanding of the implications of physical warmth for mediated social interaction. Moreover, we aimed at disentangling effects of social warmth (body temperature) from effects of non-social warmth (artificial heat sources and ambient temperature). Except for an increase in perceptions of metaphorical warmth as a consequence of higher ambient temperature, no effects were found. We use our study to pinpoint the caveats and challenges that research into warmth in affective mediated interaction faces.

We present "Nakama": A communication device that supports affective communication between a child and its - geographically separated - parent. Nakama consists of a control unit at the parent's end and an actuated teddy bear for the child. The bear contains several communication channels, including social touch, temperature, and vibrotactile heartbeats; all aimed at increasing the sense of presence. The current version of Nakama is suitable for user evaluations in lab settings, with which we aim to gain a more thorough understanding of the opportunities and limitations of these less traditional communication channels.

Since children (5-9 years old) are still developing their emotional and social skills, their social interactional behaviors in small groups might differ from adults' interactional behaviors. In order to develop a robot that is able to support children performing collaborative tasks in small groups, it is necessary to gain a better understanding of how children interact with each other. We were interested in investigating vocal turn-taking patterns as we expect these to reveal relations to collaborative and conflict behaviors, especially with children behaviors as previous literature suggests. To that end, we collected an audiovisual corpus of children performing collaborative tasks together in groups of three. Through automatic turn-taking analyses, our results showed that speaker changes with overlaps are more common than without overlaps and children seemed to show smoother turn-taking patterns, i.e., less frequent and longer lasting speaker changes, during collaborative than conflict behaviors.

When a mobile robot interacts with a group of people, it has to consider its position and orientation. We introduce a novel study aimed at generating hypotheses on suitable behavior for such social positioning, explicitly focusing on interaction with small groups of users and allowing for the temporal and social dynamics inherent in most interactions. In particular, the interactions we look at are approach, converse and retreat. In this study, groups of three participants and a telepresence robot (controlled remotely by a fourth participant) solved a task together while we collected quantitative and qualitative data, including tracking of positioning/orientation and ratings of the behaviors used. In the data we observed a variety of patterns that can be extrapolated to hypotheses using inductive reasoning. One such pattern/hypothesis is that a (telepresence) robot could pass through a group when retreating, without this affecting how comfortable that retreat is for the group members. Another is that a group will rate the position/orientation of a (telepresence) robot as more comfortable when it is aimed more at the center of that group.

This paper summarizes the contributions to the Workshop "Roadmapping the Future of Multimodal Interaction Research including Business Opportunities and Challenges". We present major challenges and ideas for making progress in the field of social signal processing and related fields as presented by the contributors of the workshop.

Touch behavior is of great importance during social interaction. To transfer the tactile modality from interpersonal interaction to other areas such as Human-Robot Interaction (HRI) and remote communication automatic recognition of social touch is necessary. This paper introduces CoST: Corpus of Social Touch, a collection containing 7805 instances of 14 different social touch gestures. The gestures were performed in three variations: gentle, normal and rough, on a sensor grid wrapped around a mannequin arm. Recognition of the rough variations of these 14 gesture classes using Bayesian classifiers and Support Vector Machines (SVMs) resulted in an overall accuracy of 54% and 53%, respectively. Furthermore, this paper provides more insight into the challenges of automatic recognition of social touch gestures, including which gestures can be recognized more easily and which are more difficult to recognize.

In this paper, we present a brief summary of the international workshop on Modeling Multiparty, Multimodal Interactions. The UM3I 2014 workshop is held in conjunction with the ICMI 2014 conference. The workshop will highlight recent developments and adopted methodologies in the analysis and modeling of multiparty and multimodal interactions, the design and implementation principles of related human-machine interfaces, as well as the identification of potential limitations and ways of overcoming them.

Sighs are non-verbal vocalisations that can carry important information about a speaker’s emotional (and psychological) state. Although sighs are commonly associated with negative emotions (e.g. giving up on something, ‘a sigh of despair’, sad-ness), sighs can also be associated with positive emotions such as relief. In order to gain a better understanding of sighing as a social and affective signal in dialogue, and to advance towards an automatic classification and interpretation of the emotional content of sighs, it is necessary to learn more about the various phonetic characteristics of sighs. To that end, we developed an annotation scheme for sighs that takes the variation in phonetic form into account. Using this scheme, an oral history corpus containing emotionally-coloured dialogues was annotated for sighs. Results show that sighs can be annotated with a suffi-cient level of reliability (Cohen’s Kappa of 0.713), and that in-deed, various types of sighs can be identified as well (Cohen’s Kappa between 0.637 and 0.805). Through a preliminary anal-ysis of emotional content words, indications were found that certain types of sighs can be associated with specific emotional contexts. Index Terms: sigh, non-verbal vocalisation, dialogue, social signal processing, annotation, oral history

The main aim of this first workshop on Mediated Touch and Affect (MeTA) is to bring together researchers from diverse communities, such as affective computing, hap tics, augmented reality, communication, design, psychology, human-robot interaction, and telepresence. The goal is to discuss the current state of research on mediated touch and affect and to formulate a research agenda for future directions in research on aspects of the touch-technology-affect triangle.

Touch is an important modality for affective communication between individuals. Here we present a system, named the TaSST (Tactile Sleeve for Social Touch), that allows two people to communicate different types of touch at a distance by touching their own forearm. In this paper we will introduce the TaSST and present a demo setup using two TaSST sleeves.

In this paper the development process and validation of the LEMtool (Layered Emotion Measurement tool) are described. The LEMtool consists of eight images that display a cartoon figure expressing four positive and four negative emotions using facial expressions and body postures. The instrument can be used during interaction with a visual interface, such as a website, and allows participants to select elements of the interface that elicit a certain emotion. The images of the cartoon figure were submitted to a validation study, in which participants rated the recognizability of the images as specific emotions. All images were found to be recognizable above chance level. In another study, the LEMtool was used to assess visual appeal judgements of a number of web pages. The LEMtool ratings were supported by visual appeal ratings of web pages both for very brief (50 milliseconds) and for long (free-viewing) stimulus exposures. Furthermore, the instrument provided insight into the elements of the web pages that elicited the emotional responses.

In this paper we outline the design process of the TaSST (Tactile Sleeve for Social Touch), a touch-sensitive vibrotactile arm sleeve. The TaSST was designed to enable two people to communicate different types of touch over a distance. The touch-sensitive surface of the sleeve consists of a grid of 4×3 sensor compartments filled with conductive wool. Each compartment controls the vibration intensity of a vibration motor, located in a grid of 4×3 motors beneath the touch-sensitive layer. An initial evaluation of the TaSST revealed that it was mainly suitable for communicating protracted (e.g. pressing), and simple (e.g. poking) touches.

Communication with others occurs through a multitude of signals, such as speech, facial expressions, and body postures. Understudied in this regard is the way we use our sense of touch in social communication. In this paper we present the TaSST (Tactile Sleeve for Social Touch), a haptic communication device that enables two people to communicate through touch at a distance.

The goal of this paper is to analyze and model the variability in speaking styles in dyadic interactions and build a predictive algorithm for listener responses that is able to adapt to these different styles. The end result of this research will be a virtual human able to automatically respond to a human speaker with proper listener responses (e.g., head nods). Our novel speaker-adaptive prediction model is created from a corpus of dyadic interactions where speaker variability is analyzed to identify a subset of prototypical speaker styles. During a live interaction our prediction model automatically identifies the closest prototypical speaker style and predicts listener responses based on this ``communicative style". Central to our approach is the idea of ``speaker profile" which uniquely identifies each speaker and enables the matching between prototypical speakers and new speakers. The paper shows the merits of our speaker-adaptive listener response prediction model by showing improvement over a state-of-the-art approach which does not adapt to the speaker. Besides the merits of speaker-adapta-tion, our experiments highlights the importance of using multimodal features when comparing speakers to select the closest prototypical speaker style.

In this paper, we investigate prosodic alignment in task-based conversations. We use the HCRC Map Task Corpus and investigate how familiarity affects prosodic alignment and how task success is related to prosodic alignment. A variety of existing alignment measures is used and applied to our data. In particular, a windowed cross-correlation procedure, that has been used previously in visual behavior research, is applied to prosodic features. In addition, we address the issue of how to separate genuine observed alignment from alignment that is a result from random coincidental behavior. Using these measures, we find some indications of prosodic convergence and synchrony in the map task conversations. Alignment tendencies are strongest for intensity, and familiarity seems to play a role in convergence. Finally, weak evidence was found for a correlation between prosodic alignment measures and task success.

This paper describes a substantial effort to build a real-time interactive multimodal dialogue system with a focus on emotional and nonverbal interaction capabilities. The work is motivated by the aim to provide technology with competences in perceiving and producing the emotional and nonverbal behaviors required to sustain a conversational dialogue. We present the Sensitive Artificial Listener (SAL) scenario as a setting which seems particularly suited for the study of emotional and nonverbal behavior since it requires only very limited verbal understanding on the part of the machine. This scenario allows us to concentrate on nonverbal capabilities without having to address at the same time the challenges of spoken language understanding, task modeling, etc. We first report on three prototype versions of the SAL scenario in which the behavior of the Sensitive Artificial Listener characters was determined by a human operator. These prototypes served the purpose of verifying the effectiveness of the SAL scenario and allowed us to collect data required for building system components for analyzing and synthesizing the respective behaviors. We then describe the fully autonomous integrated real-time system we created, which combines incremental analysis of user behavior, dialogue management, and synthesis of speaker and listener behavior of a SAL character displayed as a virtual agent. We discuss principles that should underlie the evaluation of SAL-type systems. Since the system is designed for modularity and reuse and since it is publicly available, the SAL system has potential as a joint research tool in the affective computing research community.

Social Signal Processing is the research domain aimed at bridging the social intelligence gap between humans and machines. This paper is the first survey of the domain that jointly considers its three major aspects, namely, modeling, analysis, and synthesis of social behavior. Modeling investigates laws and principles underlying social interaction, analysis explores approaches for automatic understanding of social exchanges recorded with different sensors, and synthesis studies techniques for the generation of social behavior via various forms of embodiment. For each of the above aspects, the paper includes an extensive survey of the literature, points to the most important publicly available resources, and outlines the most fundamental challenges ahead.

This demonstration aims to showcase the recently completed SEMAINE system. The SEMAINE system is a publicly available, fully autonomous Sensitive Artificial Listeners (SAL) system that consists of virtual dialog partners based on audiovisual analysis and synthesis (see http://semaine.opendfki.de/wiki). The system runs in real-time, and combines incremental analysis of user behavior, dialog management, and synthesis of speaker and listener behavior of a SAL character, displayed as a virtual agent. The SAL characters intend to engage the user in a conversation by paying attention to the user's emotions and nonverbal expressions. The characters have their own emotionally defined personality. During an interaction, the characters attempt to create an emotional workout for the user by drawing her/him towards their dominant emotion, through a combination of verbal and nonverbal expressions.

Backchannels (BCs) are short vocal and visual listener responses that signal attention, interest, and understanding to the speaker. Previous studies have investigated BC prediction in telephone-style dialogs from prosodic cues. In contrast, we consider spontaneous face-to-face dialogs. The additional visual modality allows speaker and listener to monitor each other's attention continuously, and we hypothesize that this affects the BC-inviting cues. In this study, we investigate how gaze, in addition to prosody, can cue BCs. Moreover, we focus on the type of BC performed, with the aim to find out whether vocal and visual BCs are invited by similar cues. In contrast to telephone-style dialogs, we do not find rising/falling pitch to be a BC-inviting cue. However, in a face-to-face setting, gaze appears to cue BCs. In addition, we find that mutual gaze occurs significantly more often during visual BCs. Moreover, vocal BCs are more likely to be timed during pauses in the speaker's speech.

Brain-computer interfaces (BCI) come with a lot of issues, such as delays, bad recognition, long training times, and cumbersome hardware. Gamers are a large potential target group for this new interaction modality, but why would healthy subjects want to use it? BCI provides a combination of information and features that no other input modality can offer. But for general acceptance of this technology, usability and user experience will need to be taken into account when designing such systems. This paper discusses the consequences of applying knowledge from Human-Computer Interaction (HCI) to the design of BCI for games. The integration of HCI with BCI is illustrated by research examples and showcases, intended to take this promising technology out of the lab. Future research needs to move beyond feasibility tests, to prove that BCI is also applicable in realistic, real-world settings.

In this paper, we report the results of a field trial of a Ubicomp system called CAM that is aimed at supporting and enhancing collaboration in a design studio environment. CAM uses a mobile-tagging application which allows designers to collaboratively store relevant information onto their physical design objects in the form of messages, annotations and external web links. The purpose of our field trial was to explore the role of augmented objects in supporting and enhancing creative work. Our results show that CAM was used not only to support participants' mutual awareness and coordination but also to facilitate designers in appropriating their augmented design objects to be explorative, extendable and playful, supporting creative aspects of design work. In general, our results show how CAM transformed static design objects into 'remarkable' objects that made the creative and playful side of cooperative design visible.

In this paper, we look at how prosody can be used to automatically distinguish between different dialogue act functions and how it determines degree of speaker incipiency. We focus on the different uses of 'yeah'. Firstly, we investigate ambiguous dialogue act functions of 'yeah': 'yeah' is most frequently used as a backchannel or an assessment. Secondly, we look at the degree of speakership incipiency of 'yeah': some 'yeah' items display a greater intent of the speaker to take the floor. Classi﬿cation experiments with decision trees were performed to assess the role of prosody: we found that prosody indeed plays a role in disambiguating dialogue act functions and in determining degree of speaker incipiency of 'yeah'.

We manually designed rules for a backchannel (BC) prediction model based on pitch and pause information. In short, the model predicts a BC when there is a pause of a certain length that is preceded by a falling or rising pitch. This model was validated against the Dutch IFADV Corpus in a corpus-based evaluation method. The results showed that our model performs slightly better than another well-known rule-based BC prediction model that uses only pitch information. We observed that the length of a pause preceding a BC is one of the important features in this model, next to the duration of the pitch slope at the end of an utterance. Further, we discuss implications of a corpus-based approach to BC prediction evaluation.

An experiment was conducted to investigate whether human observers use knowledge of the differences in focus of attention in multiparty interaction to identify the speaker amongst the meeting participants. A virtual environment was used to have good stimulus control. Head orientations were displayed as the only cue for focus attention. The orientations were derived from a corpus of tracked head movements. We present some properties of the relation between head orientations and speaker--listener status, as found in the corpus. With respect to the experiment, it appears that people use knowledge of the patterns in focus of attention to distinguish the speaker from the listeners. However, the human speaker identification results were rather low. Head orientations (or focus of attention) alone do not provide a sufficient cue for reliable identification of the speaker in a multiparty setting.

One of many skills required to engage properly in a conversation is to know the appropiate use of the rules of engagement. In order to engage properly in a conversation, a virtual human or robot should, for instance, be able to know when it is being addressed or when the speaker is about to hand over the turn. The paper presents a multimodal approach to end-of-speaker-turn prediction using sequential probabilistic models (Conditional Random Fields) to learn a model from observations of real-life multi-party meetings. Although the results are not as good as expected, we provide insight into which modalities are important when taking a multimodal approach to the problem based on literature and our own results.

We provide an eclectic generic framework to understand the back and forth interactions between participants in a conversation highlighting the complexity of the actions that listeners are engaged in. Communicative actions of one participant implicate the “other” in many ways. In this paper, we try to enumerate some essential relevant dimensions of this reciprocal dependence.

In this paper we discuss an approach towards passively observing students in a driving simulator. The goal is to enhance the learning experience for students taking lessons in this simulator. To this end, a virtual driving instructor is provided with added information consisting of the gaze behavior of its student. The gaze behavior is defined by estimated head locations and orientations. The learning experience for the student is enhanced by providing added feedback to the student based on his observed behavior

This paper presents an overview of our on-going work on dialogue-act classification. Results are presented on the ICSI, switchboard, and on a selection of the AMI corpus, setting a baseline for forthcoming research. For these corpora the best accuracy scores obtained are 89.27%, 65.68% and 59.76%, respectively. We introduce a smart compression technique for feature selection and compare the performance from a subset of the AMI transcriptions with AMI-ASR output for the same subset.

Work on expressive speech synthesis has long focused on the expression of basic emotions. In recent years, however, interest in other expressive styles has been increasing. The research presented in this paper aims at the generation of a storytelling speaking style, which is suitable for storytelling applications and more in general, for applications aimed at children. Based on an analysis of human storytellers' speech, we designed and implemented a set of prosodic rules for converting "neutral" speech, as produced by a text-to-speech system, into storytelling speech. An evaluation of our storytelling speech generation system showed encouraging results.

Facial movements play an important role in interpreting spoken conversations and emotions. There are several types of movements, such as conversational signals, emotion displays, etc. We call these channels of facial movement. Realistic animation of these movements improves the realism, liveliness of the interaction between human and computers using embodied conversational agents. To date, no appropriate methods have been proposed for integrating all facial movements. We propose in this paper a scheme of combining facial movements on a 3D talking head. First, we concatenate the movements in the same channel to generate smooth transitions between adjacent movements. This combination only applies to individual muscles. The movements from all channels are then combined taking into account the resolution of possible conflicting muscles

Facial expressions play an important role in face-to-face communication. With the development of personal computers capable of rendering high quality graphics, computer facial animation has produced more and more realistic facial expressions to enrich human-computer communication. In this paper, we present a simple muscle-based 3D face model that can produce realistic facial expressions in real time. We extend Waters' (1987) muscle model to generate bulges and wrinkles and to improve the combination of multiple muscle actions. In addition, we present techniques to reduce the computation burden on the muscle model.

We developed a generic teaching environment that uses agents to support learning. An agent platform developed in our research group is used. The system is generic in the sense that the cognition core (domain model, student model and instruction model) is separated from the exercise modules and user interfaces. The architecture allows different user interfaces. An application of the environment for nurse training has been implemented.

We investigated the effects of varying eye gaze behavior of an embodied conversational agent on the quality of human-agent dialogues. In an experiment we compared three versions of an agent: one with gaze behavior that is typically found to occur in human-human dialogues, one with gaze that is fixed most of the time, and a third version with random gaze behavior. The versions were found to yield significant differences in efficiency of the dialogues and in user satisfaction, amongst others.