Researchers frequently collect position and orientation data from tracking hardware during virtual reality research studies. These data capture important information about participants' movements during experiments, but often result in large and complex datasets that can be challenging to analyze and interpret. We explore the potential of standard statistical measures that could be used to interpret position and orientation tracking data, and discuss advantages and uses of each measure. We further present three new measurement techniques - 95% range, time spent at mode, and coefficient of determination. We then evaluate the effectiveness of each statistical measure to detect differences in position and orientation on two existing data sets. We find that all of the tested measures are effective in discerning known main effects in position and orientation with varying degrees of sensitivity. This work is intended to guide researchers on future analysis and interpretation of motion tracking data sets.

Virtual reality (VR) has become an increasingly popular way for learning and training. The assessment of the amount of mental effort, or cognitive load required to perform a task, is essential to create adaptive VR experiences. In this work, we conducted a large-scale study (N=738) to collect behavioral and physiological measures under different cognitive load conditions in a VR environment, and developed a novel machine learning solution to predict cognitive load in real time. Our model predicts cognitive load as a continuous value in the range from 0 to 1, where 0 and 1 correspond to the lowest and highest reported cognitive loads across all participants. On top of the point estimation of cognitive load, our model quantifies prediction uncertainty using a prediction interval. We propose a novel dual-branch attention model to accurately predict the cognitive load. We achieve a MAE (Mean Absolute Error) of 0.11. The result indicates that, with a combination of behavioral and physiological indicators, we can reliably predict cognitive load in real-time, without calibration. To support further research, we are releasing a test dataset comprising data from 100 participants for use by researchers and developers interested in machine learning, virtual reality, learning & memory, cognition, or psychophysiology. This dataset includes recordings from multiple sensors (including pupillometry, eye-tracking, and pulse plethysmography), self-reported cognitive effort, behavioral task performance, and demographic information on the sample.

Social virtual reality is an emerging medium of communication. In this medium, a user’s avatar (virtual representation) is controlled by the tracked motion of the user’s headset and hand controllers. This tracked motion is a rich data stream that can leak characteristics of the user or can be effectively matched to previously-identified data to identify a user. To better understand the boundaries of motion data identifiability, we investigate how varying training data duration and train-test delay affects the accuracy at which a machine learning model can correctly classify user motion in a supervised learning task simulating re-identification. The dataset we use has a unique combination of a large number of participants, long duration per session, large number of sessions, and a long time span over which sessions were conducted. We find that training data duration and train-test delay affect identifiability; that minimal train-test delay leads to very high accuracy; and that train-test delay should be controlled in future experiments.

In networked virtual reality (VR), user behaviors, individual differences, and group dynamics can serve as important signals for future speech behaviors, such as who the next speaker will be and the timing of turn-taking behaviors. The ability to predict and understand these behaviors offers opportunities to provide adaptive and personalized assistance, for example helping users with varying sensory abilities navigate complex social scenes and instantiating virtual moderators with natural behaviors. In this work, we predict turn-taking behaviors using features extracted based on social dynamics literature. We discuss results from a large-scale VR classroom dataset consisting of 77 sessions and 1660 minutes of small-group social interactions collected over four weeks. In our evaluation, gradient boosting classifiers achieved the best performance, with accuracies of 0.71-0.78 AUC (area under the ROC curve) across three tasks concerning the ''what'', ''who'', and ''when'' of turn-taking behaviors. In interpreting these models, we found that group size, listener personality, speech-related behavior (e.g., time elapsed since the listener's last speech event), group visual attention (e.g., the group's head orientation towards the speaker), and the listener and previous speaker's head pitch, head y-axis position, and left hand y-axis position more saliently influenced predictions. Results suggested that these features remain reliable indicators in novel social VR settings, as prediction performance is robust over time and with groups and activities not used in the training dataset. We discuss theoretical and practical implications of the work.

We introduce Audio Personas, enabling users to “decorate” themselves with body-anchored sounds in audio augmented reality. Like outfits, makeup, and fragrances, audio personas offer an alternative yet dynamic channel to augment face-to-face interactions. For instance, one can set their audio persona as rain sounds to reflect a bad mood, bee sounds to establish personal boundaries, or a playful “woosh” sound to mimic passing by someone like a breeze. To instantiate the concept, we implemented a headphone-based prototype with multi-user tracking and audio streaming. Our preregistered in-lab study with 64 participants showed that audio personas influenced how participants formed impressions. Individuals with positive audio personas were rated as more socially attractive, more likable, and less threatening than those with negative audio personas. Our study with audio designers revealed that audio personas were preferred in public and semi-public-private spaces for managing social impressions (e.g., personality) and signaling current states (e.g., emotions).

Social Virtual Reality (VR) typically entails users interacting in real time. However, asynchronous Social VR presents the possibility of combining the convenience of asynchronous communication with the high presence of VR. Because the tools to easily record and replay VR social interactions are fairly new, scholars have not yet examined how users perceive asynchronous VR social interactions, and how nonverbal transformations of recorded interactions influence user behavior. In this work, we study nonverbal transformations of group interactions around proxemics and gaze and present results from an exploratory user study (N=128) investigating their effects. We found that the combination of spatial accommodation and added gaze increases social presence, perceived attention, and mutual gaze. Results also showed an inverse relationship between interpersonal distance and perceived levels of dominance and threat of the recorded group. Finally, we outline implications for educators and virtual meeting organizers to incorporate these transformations into real-world scenarios.

While one’s facial expression and voice can be easily broadcasted from one to many via digital media, the sense of touch is limited to direct interactions. What happens if such body sensations can be shared across individuals, in which one feels a touch while watching someone else being touched? In this work, we investigated the impact of such shared body sensations on social interactions in virtual reality (VR). Building upon previous research that used psychophysics methods, our work explores the practical implications of shared body sensations in Social VR, which enables interactions beyond what’s physically possible. We conducted a withingroup user study ($\mathrm{n}=32$) in which participants observed conversations between two virtual agents and shared touch with one of the agents, as shown in Figure 1. Our results showed that even experiencing shared touch sensations several times during a conversation can affect social perception and behavior. Participants reported a stronger body illusion and empathy towards the virtual agent they shared touch with and stood closer to them. These results occurred both with and without a virtual mirror that made participants’ selfavatars more salient. The findings from this study introduce a new technique to enhance social connectedness in VR, and we discuss its applications in various contexts, such as asynchronous communication and collaboration.

Scholars who study nonverbal behavior have focused an incredible amount of work on proxemics, how close people stand to one another, and mutual gaze, whether or not they are looking at one another. Moreover, many studies have demonstrated a correlation between gaze and distance, and so-called equilibrium theory posits that people modulate gaze and distance to maintain proper levels of nonverbal intimacy. Virtual reality scholars have also focused on these two constructs, both for theoretical reasons, as distance and gaze are often used as proxies for psychological constructs such as social presence, and for methodological reasons, as head orientation and body position are automatically produced by most VR tracking systems. However, to date, the studies of distance and gaze in VR have largely been conducted in laboratory settings, observing behavior of a small number of participants for short periods of time. In this experimental field study, we analyze the proxemics and gaze of 232 participants over two experimental studies who each contributed up to about 240 minutes of tracking data during eight weekly 30-minute social virtual reality sessions. Participants' non-verbal behaviors changed in conjunction with context manipulations and over time. Interpersonal distance increased with the size of the virtual room; and both mutual gaze and interpersonal distance increased over time. Overall, participants oriented their heads toward the center of walls rather than to corners of rectangularly-aligned environments. Finally, statistical models demonstrated that individual differences matter, with pairs and groups maintaining more consistent differences over time than would be predicted by chance. Implications for theory and practice are discussed.

Social priming is the idea that observations of a virtual human (VH) engaged in short social interactions with a real or virtual human bystander can positively influence users’ subsequent interactions with that VH. In this paper we investigate the question of whether the positive effects of social priming are limited to interactions with humanoid entities. For instance, virtual dogs offer an attractive candidate for non-humanoid entities, as previous research suggests multiple positive effects. In particular, real human dog owners receive more positive attention from strangers than non-dog owners.To examine the influence of such social priming we carried out a human-subjects experiment with four conditions: three social priming conditions where a participant initially observed a VH interacting with one of three virtual entities (another VH, a virtual pet dog, or a virtual personal robot), and a non-social priming condition where a VH (alone) was intently looking at her phone as if reading something. We recruited 24 participants and conducted a mixed-methods analysis. We found that a VH’s prior social interactions with another VH and a virtual dog significantly increased participants’ perceptions of the VHs’ affective attraction. Also, participants felt more inclined to interact with the VH in the future in all of the social priming conditions. Qualitatively, we found that the social priming conditions resulted in a more positive user experience than the non-social priming condition. Also, the virtual dog and the virtual robot were perceived as a source of positive surprise, with participants appreciating the non-humanoid interactions for various reasons, such as the avoidance of social anxieties sometimes associated with humans.

As the public use of virtual reality (VR) scales, understanding how users engage across various sources of VR content is critical. 360-video is popular due to its ease of both creation and access. There are, however, few studies of 360-videos, and they suffer from three limitations. First, most studies rely on small and homogeneous samples of participants. Second, they tend to examine only a single 360-video, or a handful of them in a few exceptional cases. Third, very few studies trace participants’ VR use over multiple experiences. The current study examined a large sample of participants (511) and a large set of 360-videos (80). Each participant experienced 5 of the videos, and we tracked head movement in addition to self-report data on presence, arousal, simulator sickness, and future use intention for each video. This design allowed us to answer novel questions relating to individual differences of participants and changes in experience over time, and in general to present results of VR use at a scale not seen before in the literature. Moreover, the results suggest that looking at patterns across stimuli provide unique insights which are missed when looking only within a single piece of content.

Synchrony, the natural time-dependence of behavior in human interaction, is a pervasive feature of communication. However, most studies of synchrony have focused on dyadic interaction. In the current work, we explore synchrony in three-person teams using immersive virtual reality. Participants spent about two hours collaborating on four separate design tasks. The tracking data from the VR system allowed precise measurement of head and hand movements, facilitating calculation of synchrony. Results replicated previous work that found nonverbal synchrony in dyads in immersive VR. Moreover, we manipulated the context of the task environment, an informal garage or a traditional conference room. The environment for the task influenced synchrony, with higher levels occurring in the conference room than the garage. We also explored different methods of extending synchrony from dyads to triads, and explore the relationship of synchrony to turn taking and gaze. This paper provides theoretical insights about nonverbal synchrony and how design work functions in triads and provides suggestions for designers of VR to support good collaboration.

360-degree experiences such as cinematic virtual reality and 360-degree videos are becoming increasingly popular. In most examples, viewers can freely explore the content by changing their orientation. However, in some cases, this increased freedom may lead to viewers missing important events within such experiences. Thus, a recent research thrust has focused on studying mechanisms for guiding viewers' attention while maintaining their sense of presence and fostering a positive user experience. One approach is the utilization of diegetic mechanisms, characterized by an internal consistency with respect to the narrative and the environment, for attention guidance. While such mechanisms are highly attractive, their uses and potential implementations are still not well understood. Additionally, acknowledging the user in 360-degree experiences has been linked to a higher sense of presence and connection. However, less is known when acknowledging behaviors are carried out by attention guiding mechanisms. To close these gaps, we conducted a within-subjects user study with five conditions of no guide and virtual arrows, birds, dogs, and dogs that acknowledge the user and the environment. Through our mixed-methods analysis, we found that the diegetic virtual animals resulted in a more positive user experience, all of which were at least as effective as the non-diegetic arrow in guiding users towards target events. The acknowledging dog received the most positive responses from our participants in terms of preference and user experience and significantly improved their sense of presence compared to the non-diegetic arrow. Lastly, three themes emerged from a qualitative analysis of our participants' feedback, indicating the importance of the guide's blending in, its acknowledging behavior, and participants' positive associations as the main factors for our participants' preferences.

Domestic animals have a long history of enriching human lives physically and mentally by filling a variety of different roles, such as service animals, emotional support animals, companions, and pets. Despite this, technological realizations of such animals in augmented reality (AR) are largely underexplored in terms of their behavior and interactions as well as effects they might have on human users' perception or behavior. In this paper, we describe a simulated virtual companion animal, in the form of a dog, in a shared AR space. We investigated its effects on participants' perception and behavior, including locomotion related to proxemics, with respect to their AR dog and other real people in the environment. We conducted a 2 by 2 mixed factorial human-subject study, in which we varied (i) the AR dog's awareness and behavior with respect to other people in the physical environment and (ii) the awareness and behavior of those people with respect to the AR dog. Our results show that having an AR companion dog changes participants' locomotion behavior, proxemics, and social interaction with other people who can or can not see the AR dog. We also show that the AR dog's simulated awareness and behaviors have an impact on participants' perception, including co-presence, animalism, perceived physicality, and dog's perceived awareness of the participant and environment. We discuss our findings and present insights and implications for the realization of effective AR animal companions.

The use of virtual reality (VR) to depict climate change impacts is a popular strategy used by environmental, news, and political organizations to encourage pro-environmental outcomes. However, despite widespread dissemination of immersive content, climate change mitigation efforts remain tepid. In response, we present a VR simulation conveying the adverse effects of climate change in a personally-relevant fashion. The “Virtual Seafood Buffet” experience allows users to select from dozens of lifelike virtual seafood items and experience the degradation of that particular species based on projected climate change impacts. The developed simulation is proposed as an intervention designed to encourage climate change mitigation efforts. An overview of the simulation, its purpose, and directions for future research are outlined herein.

Intelligent Virtual Agents (IVAs) are becoming part of our everyday life, thanks to artificial intelligence technology and Internet of Things devices. For example, users can control their connected home appliances through natural voice commands to the IVA. However, most current-state commercial IVAs, such as Amazon Alexa, mainly focus on voice commands and voice feedback, and lack the ability to provide non-verbal cues which are an important part of social interaction. Augmented Reality (AR) has the potential to overcome this challenge by providing a visual embodiment of the IVA. In this paper we investigate how visual embodiment and social behaviors influence the perception of the IVA. We hypothesize that a user's confidence in an IVA's ability to perform tasks is improved when imbuing the agent with a human body and social behaviors compared to the agent solely depending on voice feedback. In other words, an agent's embodied gesture and locomotion behavior exhibiting awareness of the surrounding real world or exerting influence over the environment can improve the perceived social presence with and confidence in the agent. We present a human-subject study, in which we evaluated the hypothesis and compared different forms of IVAs with speech, gesturing, and locomotion behaviors in an interactive AR scenario. The results show support for the hypothesis with measures of confidence, trust, and social presence. We discuss implications for future developments in the field of IVAs.

Abstract In augmented reality, people can feel the illusion of virtual humans (VHs) integrated into a real (physical) space. However, affordances of the real world and virtual contents might conflict, for example, when the VHs and real objects “collide” by occupying the same space. This implausible conflict can cause a break in presence in real–virtual human interactions. In this paper, we address an effort to avoid this conflict by maintaining the VH's spatial and behavioral coherence with respect to the physical objects or events (e.g., natural occlusions and appropriate help‐requesting behaviors to avoid implausible physical–virtual collisions). We present a human subject experiment examining the effects of the physical–virtual coherence on human perceptions, such as social/copresence and behaviors with the VH. The basic ideas, experimental design, and results supporting the benefit of the VH's spatial and behavioral coherence are presented and discussed.

We explore whether a peripheral observation of apparent mutual social presence between a real human (RH) and a virtual human (VH) can in turn increase a subject's sense of social presence with the VH. In other words, we explore whether social presence can “transfer” from one RH-VH interaction to another. Specifically, we carried out an experiment where human subjects were asked to play a game with a VH. As they entered the game room, approximately half of the subjects were exposed to a brief but apparently engaging conversation between an RH and the VH. The subjects who were exposed to the brief RH-VH interaction had significantly higher measures of both emotional connection and the attentional allocation dimension of social presence for the VH, compared to those who were not. We describe the motivation, the experiment, and the results.

While performing everyday interactions, we often incidentally touch and move objects in subtle ways. These objects are not necessarily directly related to the task at hand, and the movement of an object might even be entirely unintentional. If another person is touching the object at the same time, the movement can transfer through the object and be experienced — however subtly — by the other person. For example, when one person hands a drink to another, at some point both individuals will be touching the glass, and consequently exerting small (often unnoticed) forces on the other person. Despite the frequency of such subtle incidental movements of shared objects in everyday interactions, few have examined how these movements affect human-virtual human (VH) interaction. We ran an experiment to assess how presence and social presence are affected when a person experiences subtle, incidental movement through a shared real-virtual object. We constructed a real-virtual room with a table that spanned the boundary between the real and virtual environments. The participant was seated on the real side of the table, which visually extended into the virtual world via a projection screen, and the VH was seated on the virtual side of the table. The two interacted by playing a game of “Twenty Questions,” where one player asked the other a series of 20 yes/no questions to deduce what object the other player was thinking about. During the game, the “wobbly” group of subjects experienced subtle incidental movements of the real-virtual table: the entire real-virtual table tilted slightly away/toward the subject when the virtual/real human leaned on it. The control group also played the same game, except the table did not wobble. Results indicate that the wobbly group had higher presence and social presence with the virtual human in general, with statistically significant increases in presence, co-presence, and attentional allocation. We present the experiment and results, and discuss some potential implications for virtual human systems and some potential future studies.

While there have been hundreds of psychological studies using virtual reality (VR) over the past few decades, those studies have almost exclusively been conducted in laboratory settings using small samples of college students with little demographic variance. Hence, the generalizability of the results is limited, as not all findings will apply outside the college demographic. In this paper, we present our mobile VR project (Immersion at Scale) where we conduct VR experiment sessions in naturalistic settings (e.g., local events, museums, etc.). On average, we were able to collect data from 20-25 people for each 4-hour data collection session of Immersion at Scale. We discovered a number of obstacles and opportunities based on bringing VR out into the field. Thus, we do not focus on experimental stimuli and results, but methodological guidelines based on our iterative design improvements from pilot testing.

Existing research has investigated whether virtual representations perceived to be controlled by humans (i.e., avatars) or those perceived to be controlled by computer algorithms (i.e., agents) are more influential. A meta-analysis (N = 32) examined the model of social influence in virtual environments (Blascovich, 2002) and investigated whether agents and avatars in virtual environments elicit different levels of social influence. Results indicated that perceived avatars produced stronger responses than perceived agents. Level of immersion (desktop vs. fully immersive), dependent variable type (subjective vs. objective), task type (competitive vs. cooperative vs. neutral), and actual control of the representation (human vs. computer) were examined as moderators. An interaction effect revealed that studies conducted on a desktop that used objective measures showed a stronger effect for agency than those that were conducted on a desktop but used subjective measures. Competitive and cooperative tasks showed greater agency effects than neutral tasks. Studies in which both conditions were actually human controlled showed greater agency effects than studies in which both conditions were actually computer controlled. We discuss theoretical and design implications for human–computer interaction and computer-mediated communication.

Nonverbal behavior can reveal the psychological states of those engaged in interpersonal interaction. Previous research has highlighted the relationship between gesture and learning during instruction. In the current study we applied readily available computer vision hardware and machine learning algorithms to the gestures of teacher/student dyads (N = 106) during a learning session to automatically distinguish between high and low success learning interactions, operationalized by recall for information presented during that learning session. Models predicted learning performance of the dyad with accuracies as high as 85.7 percent when tested on dyads not included in the training set. In addition, correlations were found between summed measures of body movement and learning score. We discuss theoretical and applied implications for learning.

Cyberspace technology often grants us (or others) control over our self-representations. At the click of a button, one can alter our avatars' appearance and behavior. Indeed, in virtual reality we can often appear to others as ideal in stature and weight, what ever we want in terms of age and gender, and exhibit perfect form while surfing a forty foot wave. Centuries of philosophical discussion and decades of social science research has explored the concept of “the self”, but in the digital age we are encountering iden-tity-bending only imagined by science fiction authors. In this talk, I explore a research program that explores what William Gibson referred to as “the infinite plasticity” of digital identity. In particular, I address two research areas. The first, called The Proteus Effect, explores the consequences of choosing avatars whose lappearancel differs from our own. Over forty years ago, social psychologists demonstrated self perception effects, for example wearing a black uniform causes more aggressive behavior. Similarly, as we choose our avatars online, do our avatars change us in turn? A series of studies explore how putting people in avatars of different attractiveness, height, and age alter not only behavior online but also subsequent actions in the physical world. The second area examines the consequences of choosing avatars whose Ibehaviorl differs from our own, specifically the phenomenon of seeing oneself in the third person performing an action one has never physically performed. Once a three-dimensional model resembling a specific person has been constructed, that model can be animated to perform any action fathomable to programmers. A series of studies examine how watching one's own self behave in novel manners affects memory, health behavior, and persuasion. I discuss related communication and psychological theories, as well as implications for citizens living in the digital age.

Control errors often occur in repetitive and monotonous tasks, such as manual assembly tasks. Much research has been done in the area of human error identification; however, most existing systems focus solely on the prediction of errors, not on increasing worker accuracy. The current study examines force responses before, during and after errors in a simulated assembly line task in order to determine an optimal feedback system for error reduction. Confirming previous findings, enhanced movement speed and reduced force occurred before erroneous trials and slowing occurred after erroneous trials. Given the results, we suggest a haptic feedback system which stimulates users to exert increased force levels after completing an erroneous task in order to increase degree of control and re-build worker confidence and thereby reduce overall error rate.

Summary form only given. Cyberspace technology often grants us (or others) control over our self-representations. At the click of a button, one can alter our avatars' appearance and behavior. Indeed, in virtual reality we can often appear to others as ideal in stature and weight, what ever we want in terms of age and gender, and exhibit perfect form while surfing a forty foot wave. Centuries of philosophical discussion and decades of social science research has explored the concept of “the self”, but in the digital age we are encountering identitybending only imagined by science fiction authors. In this talk, I explore a research program that explores what William Gibson referred to as “the infinite plasticity” of digital identity. In particular, I address two research areas. The first, called The Proteus Effect, explores the consequences of choosing avatars whose /appearance/ differs from our own. Over forty years ago, social psychologists demonstrated self perception effects, for example wearing a black uniform causes more aggressive behavior. Similarly, as we choose our avatars online, do our avatars change us in turn? A series of studies explore how putting people in avatars of different attractiveness, height, and age alter not only behavior online but also subsequent actions in the physical world. The second area examines the consequences of choosing avatars whose /behavior/ differs from our own, specifically the phenomenon of seeing oneself in the third person performing an action one has never physically performed. Once a three-dimensional model resembling a specific person has been constructed, that model can be animated to perform any action fathomable to programmers. A series of studies examine how watching one's own self behave in novel manners affects memory, health behavior, and persuasion. I discuss related communication and psychological theories, as well as implications for citizens living in the digital age.

Despite a variety of new communication technologies, loneliness is prevalent in Western countries. Boosting emotional communication through intimate connections has the potential to reduce loneliness. New technologies might exploit biosignals as intimate emotional cues because of their strong relationship to emotions. Through two studies, we investigate the possibilities of heartbeat communication as an intimate cue. In the first study (N = 32), we demonstrate, using self-report and behavioral tracking in an immersive virtual environment, that heartbeat perception influences social behavior in a similar manner as traditional intimate signals such as gaze and interpersonal distance. In the second study (N = 34), we demonstrate that a sound of the heartbeat is not sufficient to cause the effect; the stimulus must be attributed to the conversational partner in order to have influence. Together, these results show that heartbeat communication is a promising way to increase intimacy. Implications and possibilities for applications are discussed.

We investigated the effects of facial similarity between users and embodied agents under different experimental conditions. Sixty-four undergraduates interacted with two different embodied agents: in one case the agent was designed to look somewhat similar to the user, and in the other case the agent was designed to look dissimilar. We varied between subjects how helpful the agent was for a given task. Results showed that the facial similarity manipulation sometimes affected participants' responses, even though they did not consciously detect the similarity. Specifically, when the agent was helpful, facial similarity increased participantsi ratings of involvement. However, when exposed to unhelpful agents, male participants had negative responses to the similar-looking agent compared to the dissimilar one. These results suggest that using facially similar embodied agents has a potential large downside if that embodied agent is perceived to be unhelpful.

The use of embodied agents, defined as visual human-like representations accompanying a computer interface, is becoming prevalent in applications ranging from educational software to advertisements. In the current work, we assimilate previous empirical studies which compare interfaces with visually embodied agents to interfaces without agents, both using an informal, descriptive technique based on experimental results (46 studies) as well as a formal statistical meta-analysis (25 studies). Results revealed significantly larger effect sizes when analyzing subjective responses (i.e., questionnaire ratings, interviews) than when analyzing behavioral responses such as task performance and memory. Furthermore, the effects of adding an agent to an interface are larger than the effects of animating an agent to behave more realistically. However, the overall effect sizes were quite small (e.g., across studies, adding a face to an interface only explains approximately 2.5% of the variance in results). We discuss the implications for both designers building interfaces as well as social scientists designing experiments to evaluate those interfaces.

Understanding human-human interaction is fundamental to the long-term pursuit of powerful and natural multimodal interfaces. Nonverbal communication, including body posture, gesture, facial expression, and eye gaze, is an important aspect of human-human interaction. We introduce a paradigm for studying multimodal and nonverbal communication in collaborative virtual environments (CVEs) called Transformed Social Interaction (TSI), in which a user's visual representation is rendered in a way that strategically filters selected communication behaviors in order to change the nature of a social interaction. To achieve this, TSI must employ technology to detect, recognize, and manipulate behaviors of interest, such as facial expressions, gestures, and eye gaze. In [13] we presented a TSI experiment called non-zero-sum gaze (NZSG) to determine the effect of manipulated eye gaze on persuasion in a small group setting. Eye gaze was manipulated so that each participant in a three-person CVE received eye gaze from a presenter that was normal, less than normal, or greater than normal. We review this experiment and discuss the implications of TSI for multimodal interfaces.

Abstract Non‐verbal behaviour, particularly gaze direction, plays a crucial function in regulating conversations and providing critical social information. In the current set of studies, we represented interactants in a shared immersive virtual environment. Interactants sat in physically remote rooms, entered a common virtual room and played games of 20 questions. The interactants were represented by one of three types of avatars: (1) human forms with head movements rendered in real time; (2) human forms without head movements rendered; or (3) human voice only (i.e., a conference call). The data demonstrated that interactants in the rendered head movement condition rated a higher level of co‐presence, liked each other more, looked at each other's heads more, and spoke for a lower percentage of time during the game, compared to the other two conditions. We discuss implications for the design of shared virtual environments, the study of non‐verbal behaviour and the goal of facilitating efficient task performance. Copyright © 2002 John Wiley & Sons, Ltd.