Existing methods for human parsing into body parts and clothing often use fixed mask categories with broad labels that obscure fine-grained clothing types. Recent open-vocabulary segmentation approaches leverage pretrained text-to-image (T2I) diffusion model features for strong zero-shot transfer, but typically group entire humans into a single person category, failing to distinguish diverse clothing or detailed body parts. To address this, we propose Spectrum, a unified network for part-level pixel parsing (body parts and clothing) and instance-level grouping. While diffusion-based open-vocabulary models generalize well across tasks, their internal representations are not specialized for detailed human parsing. We observe that, unlike diffusion models with broad representations, image-driven 3D texture generators maintain faithful correspondence to input images, enabling stronger representations for parsing diverse clothing and body parts. Spectrum introduces a novel repurposing of an Image-to-Texture (I2Tx) diffusion model—obtained by fine-tuning a T2I model on 3D human texture maps—for improved alignment with body parts and clothing. From an input image, we extract human-part internal features via the I2Tx diffusion model and generate semantically valid masks aligned to diverse clothing categories through prompt-guided grounding. Once trained, Spectrum produces semantic segmentation maps for every visible body part and clothing category, ignoring standalone garments or irrelevant objects, for any number of humans in the scene. We conduct extensive cross-dataset experiments—separately assessing body parts, clothing parts, unseen clothing categories, and full-body masks—and demonstrate that Spectrum consistently outperforms baseline methods in prompt-based segmentation.

This cross-cultural study$(\mathrm{N}=108)$examines how cultural differences between Japan and Sweden influence participants social behaviors and perceptions when joining a free-standing group of two agents. Agents within the group, embodied as humans, robots, and virtual characters, respectively, use three distinct behaviors, varying with respect to politeness strategy, to request the participant to join on a specific side and position in the group. The experimental results showed that Japanese participants, from a culture characterized by higher power distance, masculinity, uncertainty avoidance, long-term orientation, restraint, and collectivism, were more likely to comply with the agent's request regarding the joining position, compared to Swedish participants. This trend was even more pronounced when comparing different types of embodiment: Japanese participants more strictly complied with human agents than with non-human agents. Additionally, Japanese females and Swedish males adhered more to social norms by avoiding walking between group members (i.e. through the group's o-space) when joining. Second, cultural differences also significantly impacted the perception of agents' politeness behaviors, while the effect of embodiment on feelings of friendliness and closeness varied depending on the culture. We reflect on our results as a basis for highlighting key challenges involved in the design of culturally adapted agents and their behaviors toward enhancing the localization of human-agent interaction.

The creation of virtual humans increasingly leverages automated synthesis of speech and gestures, enabling expressive, adaptable agents that effectively engage users. However, the independent development of voice and gesture generation technologies, alongside the growing popularity of virtual reality (VR), presents significant questions about the integration of these signals and their ability to convey emotional detail in immersive environments. In this paper, we evaluate the influence of real and synthetic gestures and speech, alongside varying levels of immersion (VR vs. 2D displays) and emotional contexts (positive, neutral, negative) on user perceptions. We investigate how immersion affects the perceived match between gestures and speech and the impact on key aspects of user experience, including emotional and empathetic responses and the sense of co-presence. Our findings indicate that while VR enhances the perception of natural gesture–voice pairings, it does not similarly improve synthetic ones—amplifying the perceptual gap between them. These results highlight the need to reassess gesture appropriateness and refine AI-driven synthesis for immersive environments.

Politeness and embodiment are pivotal elements in human-agent interactions. While many previous works advocate the positive role of embodiment in enhancing these interactions, it remains unclear how embodiment and politeness affect individuals joining groups. In this paper, we explore how politeness behaviors (verbal and nonverbal) exhibited by three distinct embodiments (humans, robots, and virtual characters) influence individuals’ decisions to join a group of two agents in a controlled experiment (N=54). We assessed agent effectiveness regarding persuasiveness, perceived politeness, and participants’ trajectories when joining the group. We found that embodiment does not significantly impact agent persuasiveness and perceived politeness, but politeness does. Direct and explicit politeness strategies have a higher success rate in persuading participants to join the group at the furthest side. Lastly, participants adhered to social norms when joining at the furthest side, maintained a greater physical distance from humans, chose longer paths, and walked faster when interacting with humans.

Existing methods for synthesizing 3D human gestures from speech have shown promising results, but they do not explicitly model the impact of emotions on the generated gestures. Instead, these methods directly output animations from speech without control over the expressed emotion. To address this limitation, we present AMUSE, an emotional speech-driven body animation model based on latent diffusion. Our observation is that content (i.e., gestures related to speech rhythm and word utterances), emotion, and personal style are separable. To account for this, AMUSE maps the driving audio to three disentangled latent vectors: one for content, one for emotion, and one for personal style. A latent diffusion model, trained to generate gesture motion sequences, is then conditioned on these latent vectors. Once trained, AMUSE synthesizes 3D human gestures directly from speech with control over the expressed emotions and style by combining the content from the driving speech with the emotion and style of another speech sequence. Randomly sampling the noise of the diffusion model further generates variations of the gesture with the same emotional expressivity. Qualitative, quantitative, and perceptual evaluations demonstrate that AMUSE outputs realistic gesture sequences. Compared to the state of the art, the generated gestures are better synchronized with the speech content, and better represent the emotion expressed by the input speech. Our code is available at amuse.is.tue.mpg.de.

Politeness is at the core of the common set of behavioral norms that regulate human communication and is therefore of significant interest in the design of Human-Robot Interactions. In this paper, we investigate how the politeness behaviors of a humanoid robot impact human decisions about where to join a group of two robots. We also evaluate the resulting impact on the perception of the robot's politeness. In a study involving 59 participants, the main (Pepper) robot in the group invited participants to join using six politeness behaviors derived from Brown and Levinson's politeness theory. It requests participants to join the group at the furthest side of the group which involves more effort to reach than a closer side that is also available to the participant but would ignore the request of the robot. We evaluated the robot's effectiveness in terms of persuasiveness, politeness, and clarity. We found that more direct and explicit politeness strategies derived from the theory have a higher level of success in persuading participants to join at the furthest side of the group. We also evaluated participants' adherence to social norms i.e. not walking through the center, or o-space, of the group when joining it. Our results showed that participants tended to adhere to social norms when joining at the furthest side by not walking through the center of the group of robots, even though they were informed that the robots were fully automated.

Multimodal communication is essential in human interactions, as it allows for a more comprehensive and nuanced exchange of information and emotions. The use of multiple communication channels such as speech, body language, and gaze can enhance the clarity and richness of the communication, leading to better understanding and more effective social interactions. This paper investigates the importance of multimodal expressive communication, specifically voice, arm gestures, and gaze, in regulating human-agent interaction when joining a group of two virtual agents in a virtual reality environment. One of the virtual agents in the group uses politeness behaviors based on Brown and Levinson's politeness theory to invite participants to join the group at the side further to them, even though a closer side is available. The study finds that a combination of all modalities (verbal, gaze, arm gesture) is more effective in persuading participants to join the group at the farthest side, and arm gestures alone are more effective than gaze behavior although they are perceived to be less polite. Furthermore, although verbal-only communication can be as persuasive as other modalities, it can place a greater cognitive load on participants. This increased cognitive load may lead to delayed responses in comparison to other modalities. The findings give insight to designers of human-agent interaction systems about the use of multiple communication channels, particularly nonverbal behaviors such as arm gestures, to enhance the effectiveness of persuasive communication but they also need to balance this with other factors such as the impression and perceived politeness of virtual agents.

While creativity has been previously studied in Child-Robot Interaction (cHRI), the effect of regulatory focus on creativity skills has not been investigated. This paper presents an exploratory study that, for the first time, uses the Regulatory Focus Theory (RFT) to assess children's creativity skills in an educational context with a social robot. We investigated whether two key emotional regulation techniques, promotion (approach) and prevention (avoidance), stimulate creativity during a story-telling activity between a child and a robot. We conducted a between-subjects field study with 69 children between the ages of 7 and 9 years old, divided between two study conditions: (1) promotion, where a social robot primes children for action by eliciting positive emotional states, and (2) prevention, where a social robot primes children for avoidance by evoking a states related to security and safety associated with blockage-oriented behaviors. To assess changes in creativity as a response to the priming interaction, children were asked to tell stories to the robot before (pre-test) and after (post-test) the priming interaction. We measured creativity levels by analyzing the verbal content of the stories. We coded verbal expressions related to creativity variables, including fluency, flexibility, elaboration, and originality. Our results show that children in the promotion condition generated significantly more ideas, and their ideas were on average more original in the stories they created in the post-test rather than in the pre-test. We also modeled the process of creativity that emerges during storytelling in response to the robot's verbal behavior. This paper enriches the scientific understanding of creativity emergence in child-robot collaborative interactions.

We present the design and evaluation of a storytelling activity between children and an autonomous robot aiming at nurturing children’s creativity. We assessed whether a robot displaying creative behavior will positively impact children’s creativity skills in a storytelling context. We developed two models for the robot to engage in the storytelling activity: creative model, where the robot generates creative story ideas, and the non-creative model, where the robot generates non-creative story ideas. We also investigated whether the type of the storytelling interaction will have an impact on children’s creativity skills. We used two types of interaction: 1) Collaborative, where the child and the robot collaborate together by taking turns to tell a story. 2) Non-collaborative: where the robot first tells a story to the child and then asks the child to tell it another story. We conducted a between-subjects study with 103 children in four different conditions: Creative collaborative, Non-creative collaborative, Creative non-collaborative and Non-Creative non-collaborative. The children’s stories were evaluated according to the four standard creativity variables: fluency, flexibility, elaboration and originality. Results emphasized that children who interacted with a creative robot showed higher creativity during the interaction than children who interacted with a non-creative robot. Nevertheless, no significant effect of the type of the interaction was found on children’s creativity skills. Our findings are significant to the Child-Robot interaction (cHRI) community since they enrich the scientific understanding of the development of child-robot encounters for educational applications.

In an attempt to nurture children's creativity, we developed a creative conversational agent to be used in a collaborative storytelling context with a child. We presented a novel approach to develop creative Artificial Intelligence (AI). Our approach uses the four creativity measures: fluency, flexi-bility, elaboration and originality in order to generate creative behavior. We analyzed and annotated our previously collected storytelling data sets -collected with children- according to our four creativity measures. We then used the extracted and annotated data (636 statements) in order to fine-tune two pre-trained language models (Open AI GPT-3). The two models were aimed at generating creative versus non-creative behavior in a collaborative storytelling scenario. We developed the two models to be able to assess the results and compare them together. We conducted an evaluation to assess stories generated collaboratively between a human and both agents separately (n = 26). Adult Users rated the creativity of the agent according to the stories generated. Results showed that the creative agent was perceived as significantly more creative than the non-creative agent. With the experiment results confirming the validity of our system, we may therefore proceed with testing the effects of the creative behavior of the agent on children's creativity skills.

When modeling life-like Embodied Conversational Agents (ECAs), conveying politeness through verbal and nonverbal behaviors with persuasive intents is a significant challenge, as it underlies the conventional set of behavioral rules that govern human communication. In the present study, we explore the adherence to such rules in the context of joining a small, freestanding conversational group of agents in VR. In particular, we focus on the behavior adopted by participants while walking towards the agents, and on whether ECAs were treated in the same way human agents normally are. 45 test subjects were invited by an ECA to walk towards the group by applying one of six possible politeness strategies; after freely joining the group, they were asked to rate the agent's politeness according to four distinct aspects (Clarity, Face loss, Positive face, and Negative face). Across all strategies, in 48% of the trials participants were successfully persuaded to join the group at an inconvenient location. Out of those trials, participants adhered to social conventions by not crossing the convex empty space between the group members (o-space) in 75% of them on average. Additionally, analysis of verbal and nonverbal behaviors in ECAs shows that direct request strategies are more effective than indirect ones, although in some cases they may be perceived as less polite.

According to psychology research, emotional induction has positive implications in many domains such as therapy and education. Our aim in this paper was to manipulate the Regulatory Focus Theory to assess its impact on the induction of regulatory focus related emotions in children in a pretend play scenario with a social robot. The Regulatory Focus Theory suggests that people follow one of two paradigms while attempting to achieve a goal; by seeking gains (promotion focus - associated with feelings of happiness) or by avoiding losses (prevention focus - associated with feelings of fear). We conducted a study with 69 school children in two different conditions (promotion vs. prevention). We succeeded in inducing happiness emotions in the promotion condition and found a resulting positive effect of the induction on children’s social engagement with the robot. We also discuss the important implications of these results in both educational and child robot interaction fields.

Creativity is a vital inherent human trait. In an attempt to stimulate children's creativity, we present the design and evaluation of an interaction between a child and a social robot in a storytelling context. Using a software interface, children were asked to collaboratively create a story with the robot. We conducted a study with 38 children in two conditions. In one condition, the children interacted with a robot exhibiting creative behavior while in the other condition, they interacted with a robot exhibiting non creative behavior. The robot's creativity was defined as verbal and performance creativity. The robot's creative and non creative behaviors were extracted from a previously collected data set and were validated in an online survey with 100 participants. Contrary to our initial hypothesis, children's creativity measures were not higher in the creative condition than in the non creative condition. Our results suggest that merely the robot's creative behavior is insufficient to stimulate creativity in children in a child robot interaction. We further discuss other design factors that may facilitate sparking creativity in children in similar settings in the future.

Politeness behaviors could affect individuals' decisions heavily in their daily lives and may therefore also play an important role in human-agent interactions. This study considers the impact of politeness behaviors made by a virtual agent, already in a small face-to-face conversational group with another agent, on a human participant as they approach to join it in a virtual environment displayed on a monitor. The agent uses five verbal and nonverbal politeness strategies, ranging from indirect and implicit to direct and explicit, in an attempt to influence the participant to join the group at an inconvenient location, which requires more time and effort than a direct route that would ignore the invitation of the agent. In addition to assessing the success of the strategies at influencing participant behavior, the participants' perception of the agent's persuasive behavior is assessed in relation to clarity, face loss, positive face, and negative face. Based on results from a within-subjects experiment with 30 participants, we found that more direct and explicit politeness strategies have a higher level of success when requesting a participant to join a small group at an inconvenient location, but sometimes negatively impact their perception of the agent. A positive politeness strategy was found to be the most effective for both persuasive success and maintaining a positive impression of the agent.

Mobile robots that approach free-standing conversational groups to join them should behave in a safe and socially-acceptable way. Existing trajectory generation methods focus on collision avoidance with pedestrians, and the models that generate approach behaviors into groups are evaluated in simulation. However, it is challenging to generate approach and join trajectories that avoid collisions with group members while also ensuring that they do not invoke feelings of discomfort. In this paper, we conducted an experiment to examine the impact of three trajectory generation methods for a mobile robot to approach groups from multiple directions: a Wizard-of-Oz (WoZ) method, a procedural social-aware navigation model (PM) and a novel generative adversarial model imitating human approach behaviors (IL). Measures also compared two camera viewpoints and static versus quasi-dynamic groups. The latter refers to a group whose members change orientation and position throughout the approach task, even though the group entity remains static in the environment. This represents a more realistic but challenging scenario for the robot. We evaluate three methods with objective measurements and subjective measurements from viewer perception, and results show that WoZ and IL have comparable performance, and both perform better than PM under most conditions.

In order to design social agents for long term interactions, it is important to enable them to adapt to the users. In this paper, we chose personality as a medium for adaptation. We conducted a study with 20 participants who watched a story presented by a virtual character in one of two conditions: extroverted or introverted. The study aimed at assessing the impacts of matching the personality of the user with the virtual character through body language on the likability of the character and the information recall of the story. Our findings do not appear to coincide with theoretical expectations since the extroverted character had higher ratings of likability regardless of the personality of the user. Results have also shown a marginal positive effect of the encounter with the introverted character in terms of memory recall. We discuss the important implications that these results may have in the future for human agent interaction design.

Goal directed agent navigation in crowd simulations involves a complex decision making process. An agent must avoid all collisions with static or dynamic obstacles (such as other agents) and keep a trajectory faithful to its target at the same time. This seemingly global optimization problem can be broken down into smaller local optimization problems by looking at a concept of criticality. Our method resolves critical agents - agents that are likely to come within collision range of each other - in order of priority using a Particle Swarm Optimization scheme. The resolution involves altering the velocities of agents to avoid criticality. Results from our method show that the navigation problem can be solved in several important test cases with minimal number of collisions and minimal deviation to the target direction. We prove the efficiency and correctness of our method by comparing it to four other well-known algorithms, and performing evaluations on them based on various quality measures.

While many works involving human-agent interactions have focused on individuals or crowds, modelling interactions on the group scale has not been considered in depth. Simulation of interactions with groups of agents is vital in many applications, enabling more comprehensive and realistic behavior encompassing all possibilities between crowd and individual levels. In this paper, we propose a novel neural network App-LSTM to generate the approach trajectory of an agent towards a small free-standing conversational group of agents. The App-LSTM model is trained on a dataset of approach behaviors towards the group. Since current publicly available datasets for these encounters are limited, we develop a social-aware navigation method as a basis for creating a semi-synthetic dataset composed of a mixture of real and simulated data representing safe and socially-acceptable approach trajectories. Via a group interaction module, App-LSTM then captures the position and orientation features of the group and refines the current state of the approaching agent iteratively to better focus on the current intention of group members. We show our App-LSTM outperforms baseline methods in generating approaching group trajectories.

Deep reinforcement learning has recently been widely applied in robotics to study tasks such as locomotion and grasping, but its application to social human-robot interaction (HRI) remains a challenge. In this paper, we present a deep learning scheme that acquires a prior model of robot approaching behavior in simulation and applies it to real-world interaction with a physical robot approaching groups of humans. The scheme, which we refer to as Staged Social Behavior Learning (SSBL), considers different stages of learning in social scenarios. We learn robot approaching behaviors towards small groups in simulation and evaluate the performance of the model using objective and subjective measures in a perceptual study and a HRI user study with human participants. Results show that our model generates more socially appropriate behavior compared to a state-of-the-art model.

Robots that navigate to approach free-standing conversational groups should do so in a safe and socially acceptable manner. This is challenging since it not only requires the robot to plot trajectories that avoid collisions with members of the group, but also to do so without making those in the group feel uncomfortable, for example, by moving too close to them or approaching them from behind. Previous trajectory prediction models focus primarily on formations of walking pedestrians, and those models that do consider approach behaviours into free-standing conversational groups typically have handcrafted features and are only evaluated via simulation methods, limiting their effectiveness. In this paper, we propose AppGAN, a novel trajectory prediction model capable of generating trajectories into free-standing conversational groups trained on a dataset of safe and socially acceptable paths. We evaluate the performance of our model with state-of-the-art trajectory prediction methods on a semi-synthetic dataset. We show that our model outperforms baselines by taking advantage of the GAN framework and our novel group interaction module.

Mixed reality offers new potentials for social interaction experiences with virtual agents. In addition, it can be used to experiment with the design of physical robots. However, while previous studies have investigated comfortable social distances between humans and artificial agents in real and virtual environments, there is little data with regards to mixed reality environments. In this paper, we conducted an experiment in which participants were asked to walk up to an agent to ask a question, in order to investigate the social distances maintained, as well as the subject's experience of the interaction. We manipulated both the embodiment of the agent (robot vs. human and virtual vs. physical) as well as closed vs. open posture of the agent. The virtual agent was displayed using a mixed reality headset. Our experiment involved 35 participants in a within-subject design. We show that, in the context of social interactions, mixed reality fares well against physical environments, and robots fare well against humans, barring a few technical challenges.

Modelling and simulation of pedestrian crowds require agents to reach pre-determined goals and avoid collisions with static obstacles and dynamic pedestrians, while maintaining natural gait behaviour. We model pedestrians as autonomous, learning, and reactive agents employing Reinforcement Learning (RL). Typical RL-based agent simulations suffer poor generalization due to handcrafted reward function to ensure realistic behaviour. In this work, we model pedestrians in a modular framework integrating navigation and collision-avoidance tasks as separate modules. Each such module consists of independent state-spaces and rewards, but with shared action-spaces. Empirical results suggest that such modular framework learning models can show satisfactory performance without tuning parameters, and we compare it with the state-of-art crowd simulation methods.

Pedestrian trajectory prediction is a challenging problem. One of the aspects that makes it so challenging is the fact that the future positions of an agent are not only determined by its previous positions, but also by the interaction of the agent with its neighbors. Previous methods, like Social Attention have considered the interactions with all agents as neighbors. However, this ends up assigning high attention weights to agents who are far away from the queried agent and/or moving in the opposite direction, even though, such agents might have little to no impact on the queried agent's trajectory. Furthermore, trajectory prediction of a queried agent involving all agents in a large crowded scenario is not efficient. In this paper, we propose a novel approach for selecting neighbors of an agent by modeling its perception as a combination of a location and a locomotion model. We demonstrate the performance of our method by comparing it with the existing state-of-the-art method on publicly available datasets. The results show that our neighbor selection model overall improves the accuracy of trajectory prediction and enables prediction in scenarios with large numbers of agents in which other methods do not scale well.

Agent-based crowd simulation in virtual environments is of great utility in a variety of domains, from the entertainment industry to serious applications including mobile robots and swarms. Many studies of crowd behavior simulations do not consider the fact that people tend to congregate in smaller social gatherings, such as friends, or families, rather than walking alone. Based on a real-time crowd simulator which has been implemented as a unilateral incompressible fluid and augmented with group behaviors, a perceptual study was conducted to determine the impact of groups on the perception of the crowds at various densities from different camera views. If it is not possible to see groups under certain circumstances, then it may not be necessary to simulate them, to reduce the amount of calculations, an important issue in real-time simulations. This study provides researchers with a proper reference to design better algorithms to simulate realistic behaviors.

Understanding emotional states is a challenging task which frequently leads to misinterpretation even in human observers. While the perception of emotions has been studied extensively in human psychology, little is known about what factors influence the human perception of emotions in robots and virtual characters. In this paper, we build on the Brunswik lens model to investigate the influence of (a) the agent's embodiment using a 2D virtual character, a 3D blended embodiment, a recording of the 3D platform and a recording of a human, as well as (b) the level of human-likeness on people's ability to interpret emotional facial expressions in an agent. In addition, we measure social traits of the human observers and analyze how they correlate to the success in recognizing emotional expressions. We find that interpersonal differences play a minor role in the perception of emotional states. However, both embodiment and human-likeness as well as related perceptual dimensions such as perceived social presence and uncanniness have an effect on the attribution of emotional states.

Mimicry plays an important role in social interaction. In human communication, it is used to establish rapport and bonding both with other humans, as well as robots and virtual characters. However, little is known about the underlying factors that elicit mimicry in humans when interacting with a robot. In this work, we study the influence of embodiment on participants' ability to mimic a social character. Participants were asked to intentionally mimic the laughing behavior of the Furhat mixed embodied robotic head and a 2D virtual version of the same character. To explore the effect of embodiment, we present two novel approaches to automatically assess people's ability to mimic based solely on videos of their facial expressions. In contrast to participants' self-assessment, the analysis of video recordings suggests a better ability to mimic when people interact with the 2D embodiment.

This paper investigates the influence of multimodal incongruent gender cues on the perception of a robot's uncanniness and gender in children. The back-projected robot head Furhat was equipped with a female and male face texture and voice synthesizer and the voice and facial cues were tested in congruent and incongruent combinations. 106 children between the age of 8 and 13 participated in the study. Results show that multimodal incongruent cues do not trigger the feeling of uncanniness in children. These results are significant as they support other recent research showing that the perception of uncanniness cannot be triggered by a categorical ambiguity in the robot. In addition, we found that children rely on auditory cues much stronger than on the facial cues when assigning a gender to the robot if presented with incongruent cues. These findings have implications for the robot design, as it seems possible to change the gender of a robot by only changing its voice without creating a feeling of uncanniness in a child.

Engagement in task orientated social robotics is a complex phenomenon, consisting of both task and social elements. Previous work in this area tends to focus on these aspects in isolation without consideration for the positive or negative effects one might cause the other. We explore both, in an attempt to understand how engagement with the task might effect the social relationship with the robot, and vice versa. In this paper, we describe the analysis of participant self-report data collected during an exploratory pilot study used to evaluate users' “perception of engagement”. We discuss how the results of our analysis suggest that ultimately, it was the users' own perception of the robots' characteristics such as friendliness, helpfulness and attentiveness which led to sustained engagement with both the task and robot.

Abstract A person's emotions and state of mind are apparent in their face and eyes. As a Latin proverb states: ‘The face is the portrait of the mind; the eyes, its informers’. This presents a significant challenge for Computer Graphics researchers who generate artificial entities that aim to replicate the movement and appearance of the human eye, which is so important in human–human interactions. This review article provides an overview of the efforts made on tackling this demanding task. As with many topics in computer graphics, a cross‐disciplinary approach is required to fully understand the workings of the eye in the transmission of information to the user. We begin with a discussion of the movement of the eyeballs, eyelids and the head from a physiological perspective and how these movements can be modelled, rendered and animated in computer graphics applications. Furthermore, we present recent research from psychology and sociology that seeks to understand higher level behaviours, such as attention and eye gaze, during the expression of emotion or during conversation. We discuss how these findings are synthesized in computer graphics and can be utilized in the domains of Human–Robot Interaction and Human–Computer Interaction for allowing humans to interact with virtual agents and other artificial entities. We conclude with a summary of guidelines for animating the eye and head from the perspective of a character animator.

Simulating the behavior of crowds of artificial entities that have humanoid embodiments has become an important element in computer graphics and special effects. However, many important questions remain in relation to the perception of social behavior and expression of emotions in virtual crowds. Specifically, few studies have considered the role of background context on the perception of the full-body emotion expressed by sub-constituents of the crowd i.e. individuals and small groups. In this paper, we present the results of perceptual studies in which animated scenes of expressive virtual crowd behavior were rated in terms of their valence by participants. The behaviors of a task-irrelevant crowd in the background were altered between neutral, happy and sad in order to investigate effects on the perception of emotion from task-relevant individuals in the foreground. Effects of the task irrelevant background on ratings of foreground characters were found, including cases that accompanied negatively valenced stimuli.

In our work we explore the development of a computational model capable of automatically detecting engagement in social human-robot interactions from real-time sensory and contextual input. However, to train the model we need to establish ground truths of engagement from a large corpus of data collected from a study involving task and social-task engagement. Here, we intend to advance the current state-of-the-art by reducing the need for unreliable post-experiment questionnaires and costly time-consuming annotation with the novel introduction of implicit probes. A non-intrusive, pervasive and embedded method of collecting informative data at different stages of an interaction.

We present a case study analyzing and discussing the effects of introducing the requirement of public outreach of original student work into the project-based learning of Advanced Graphics and Interaction (AGI) at KTH Royal Institute of Technology. We propose Expo-Based Learning as Project-Based Learning augmented with the constructively aligned goal of achieving public outreach beyond the course. We promote this outreach through three challenges: 1) large public presentations; 2) multidisciplinary collaboration; and 3) professional portfolio building. We demonstrate that the introduction of these challenges, especially the public presentations, had lasting positive impact in the intended technical learning outcomes of AGI with the added benefit of learning team work, presentation skills, timeliness, accountability, self-motivation, technical expertise, and professionalism.

This workshop covers real-time computational techniques for the recognition and interpretation of human affective and social behaviour, and techniques for synthesis of believable social behaviour supporting real-time adaptive human-agent and human-robot interaction in real-world environments.

The focus of this project is to design, develop and evaluate a new computational model for automatically detecting change in task engagement. This work will be applied to robotic tutors to enhance and support the learning experience, enabling timely pedagogical and empathic intervention. This work is intended to forward the current state of the art by 1) exploring how to automatically detect engagement with a learning task, 2) designing and developing new approaches to machine learning for adaptive platform-independent modelling and 3) evaluation of its effectiveness for building and maintaining learner engagement across different tutor embodiments, for example a physical and virtual embodiment.

Simulating vast crowds of autonomous agents within a procedurally generated virtual environment is a challenging endeavour from a technical perspective, however it becomes even more difficult when the subjective nature of perception is also taken into account. Agent behaviour is the product of artificial intelligence systems working in tandem, however the sophistication of these systems is not a guarantee of achieving believable behaviour. Within locations based upon reality such as an urban environment, the perceived realism of agent behaviour becomes even harder to achieve. This paper presents the development of a crowd simulation that is based upon a real-life urban environment, which is then subjected to perceptual experimentation to identify features of behaviour which can be linked to perceived realism. This research is predicted to feedback into the development processes of inhabited cities, especially those attempting to simulate perceptually realistic agents as it will highlight features of behaviour that are important to implement. The perceptual experimentation methodologies presented can also be adapted and potentially utilised to test other types of crowd simulation, whether it be for the purposes of computer games or even urban planning and health and safety.

Abstract Misconceptions about the nature of the computing disciplines pose a serious problem to university faculties that offer computing degrees, as students enrolling on their programmes may come to realise that their expectations are not met by reality. This frequently results in the students’ early disengagement from the subject of their degrees which in turn can lead to excessive ‘wastage’, that is, reduced retention. In this paper, we report on our academic group’s attempts within creative computing degrees at a UK university to counter these problems through the introduction of a 6 week long project that newly enrolled students embark on at the very beginning of their studies. This group project, involving the creation of a 3D etch‐a‐sketch‐like computer graphics application with a hardware interface, provides a breadth‐first, activity‐led introduction to the students’ chosen academic discipline, aiming to increase student engagement while providing a stimulating learning experience with the overall goal to increase retention. We present the methods and results of two iterations of these projects in the 2009/2010 and 2010/2011 academic years, and conclude that the approach worked well for these cohorts, with students expressing increased interest in their chosen discipline, in addition to noticeable improvements in retention following the first year of the students’ studies.

Affect-sensitive systems such as social robots and virtual agents are increasingly being investigated in real-world settings. In order to work effectively in natural environments, these systems require the ability to infer the affective and mental states of humans and to provide appropriate timely output that helps to sustain long-term interactions. This special issue, which appears in two parts, includes articles on the design of socio-emotional behaviors and expressions in robots and virtual agents and on computational approaches for the automatic recognition of social signals and affective states.

Successful human interaction commonly involves prototypical exchanges where interactors are engaged, synchronized, and harmonious in their behaviors. The copying of aspects of the other's behavior, at different levels, seems central to establishing and maintaining such empathic connections. Yet, many questions remain unanswered, particularly how it is possible to reflect the same affective content back to the other when the actual motion itself is not exactly the same as theirs. This paper presents a perceptual study in which emotional gestures conducted by an actor were mapped onto synthesized versions generated by an embodied virtual agent. Copying is at the expressive level, where qualities such as the fluidity or expansiveness of gestures are considered, rather than exact low-level motion matching. Participants were later asked to rate the emotional content of video recordings of both the original and the synthesized gestures. A statistical analysis shows that, in most cases, participants associated the emotional content of the agent's gestures with that intended to be expressed by the original actor. The results suggest that a combination of the type of movement performed and its quality is important for successfully communicating emotions.

In this article, we evaluate the effects of position, orientation, and camera viewpoint on the plausibility of pedestrian formations. In a set of three perceptual studies, we investigated how humans perceive characteristics of virtual crowds in static scenes reconstructed from annotated still images, where the orientations and positions of the individuals have been modified. We found that by applying rules based on the contextual information of the scene, we improved the perceived realism of the crowd formations when compared to random formations. We also examined the effect of camera viewpoint on the plausibility of virtual pedestrian scenes, and we found that an eye-level viewpoint is more effective for disguising random behaviors, while a canonical viewpoint results in these behaviors being perceived as less realistic than an isometric or top-down viewpoint. Results from these studies can help in the creation of virtual crowds, such as computer graphics pedestrian models or architectural scenes, and identify situations when users' perception is less accurate.

The 3rd International Workshop on Affective Interaction in Natural Environments, AFFINE, follows a number of successful AFFINE workshops and events commencing in 2008.A key aim of AFFINE is the identification and investigation of significant open issues in real-time, affect-aware applications 'in the wild' and especially in embodied interaction, for example, with robots or virtual agents. AFFINE seeks to bring together researchers working on the real-time interpretation of user behaviour with those who are concerned with social robot and virtual agent interaction frameworks.

We present a method for the creation of autonomous agent memory systems that support real-time social behaviours relating to perception and attention. Our memory design methodology is modular in nature: modules are connected by transformation operators representative of different forms of information processing and which fall into two general categories: information filters and integrators. Filters are selective and only retain relevant information, while integrators process information into more compact and high-level representations. We demonstrate the flexibility and feasibility of our methodology by elaborating two examples which have been implemented and shown successful for supporting fundamental agent social behaviours in a virtual environment.

We present a system for the automatic generation of bottom-up visual attention behaviours in virtual humans. Bottom-up attention refers to the way in which the environment solicits one's attention without regard to task-level goals. Our framework is based on the interactions of multiple components: a synthetic vision system for perceiving the virtual world, a model of bottom-up attention for early visual processing of perceived stimuli, a memory system for the storage of previously sensed data and a gaze controller for the generation of resultant behaviours. Our aim is to provide a feeling of presence in inhabited virtual environments by endowing agents with the ability to pay attention to their surroundings.

No abstract available.

Abstract Work on levels of detail for human simulation has occurred mainly on a geometrical level, either by reducing the numbers of polygons representing a virtual human, or replacing them with a two‐dimensional imposter. Approaches that reduce the complexity of motions generated have also been proposed. In this paper, we describe ongoing development of a framework for Adaptive Level Of Detail for Human Animation (ALOHA), which incorporates levels of detail for not only geometry and motion, but also includes a complexity gradient for natural behaviour, both conversational and social. ACM CSS: I.3.7 Three‐Dimensional Graphics and Realism—Animation

Abstract A memory model based on ``stage theory'', an influential concept of memory from the field of cognitive psychology,is presented for application to autonomous virtual humans. The virtual human senses external stimuli througha synthetic vision system. The vision system incorporates multiple modes of vision in order to accommodate aperceptual attention approach. The memory model is used to store perceived and attended object information atdifferent stages in a filtering process. The methods outlined in this paper have applications in any area wheresimulation‐based agents are used: training, entertainment, ergonomics and military simulations to name but afew. ACM CSS: I. 3.7 Computer Graphics‐‐Virtual reality