Communication barriers contribute low clinical trial participation, which is crucial in advancing breast cancer treatment. Patient-centered communication and considering user characteristics like health literacy may improve communication about clinical trials. Virtual agent interventions can employ these strategies to improve communication. However, it is unclear how the adoption of these strategies impact trust and use of a virtual agent intervention. Therefore, we conducted a study where 103 breast cancer patients interacted with a web-based, patient-centered virtual agent intervention to learn about clinical trials. Although the virtual agent’s communication strategy alone did not influence perceptions of trust, findings revealed that accounting for participants’ health literacy significantly impacted trust and unveiled a more nuanced understanding of patient-centered communication strategies. This work contributes to the design of virtual agent interventions, highlighting the necessity of incorporating user characteristics like health literacy when developing interventions for breast cancer patients for clinical trial engagement.

Pedagogical Conversational Agents (PCAs) are increasingly used to provide personalized support in digital learning. This study examines how different levels of learner control that a virtual teaching assistant (TA) offered impact students' self-efficacy, learning effectiveness, and satisfaction, given the moderating effects of student characteristics. A between-subjects study with 215 students across three learner control conditions revealed that the moderate control (i.e., Assisted Inquiry) led to higher perceived learning effectiveness than the high control. Agreeableness, Extraversion, and Openness were found to significantly influence students' learning. These findings underscore the need for adaptive PCA systems that tailor instructional support with scaffolding and balanced learning interventions while accounting for learners' personality traits to enhance student engagement and perceived learning effectiveness.

Figure 1: The single virtual agent (left) and multiple virtual agent (right) interfaces.

Advances in AI, including large language models, enable virtual health assistants (VHAs) to tailor messaging to cultural and individual needs.However, to fully realize this potential, VHA voices must match or exceed human voices in influencing behavioral intentions.We conducted a multi-study investigation on whether matching text-to-speech (TTS) voices to human voices across vocal traits could promote colorectal cancer (CRC) screening among Black or African American adults.Study 1 (N = 313) tested whether a TTS voice prosodically matched to a human voice could elicit equivalent screening intentions using a White male VHA.It also evaluated whether increasing vocal confidence via SSML would enhance effectiveness.A user preference study (N = 317) assessed VHA identity and delivery modality preferences, finding strong support for a Black female VHA.Building on these results, Study 2 (N = 106) replicated the TTS-human voice comparison using a Black female VHA.Across studies, matching TTS voices to human voices on frequency, speech rate, and pitch variation achieved equivalent * Both authors contributed equally to this research.

Figure 1: The virtual health assistant with example dialogue used in the stress-coping gratitude journaling intervention

Virtual humans (i.e., embodied conversational agents) have the potential to support college students’ mental health, particularly in Science, Technology, Engineering, and Mathematics (STEM) fields where students are at a heightened risk of mental disorders such as anxiety and depression. A comprehensive understanding of students, considering their cultural characteristics, experiences, and expectations, is crucial for creating timely and effective virtual human interventions. To this end, we conducted a user study with 481 computer science students from a major university in North America, exploring how they co-designed virtual humans to support mental health conversations for students similar to them . Our findings suggest that computer science students who engage in co-design processes of virtual humans tend to create agents that closely resemble them demographically—agent-designer demographic similarity. Key factors influencing virtual human design included age, gender, ethnicity, and the matching between appearance and voice. We also observed that the demographic characteristics of virtual human designers, especially ethnicity and gender, tend to be associated with those of the virtual humans they designed. Finally, we provide insights concerning the impact of user-designer demographic similarity in virtual humans’ effectiveness in promoting mental health conversations when designers’ characteristics are shared explicitly or implicitly. Understanding how virtual humans’ characteristics serve users’ experiences in mental wellness conversations and the similarity-attraction effects between agents, users, and designers may help tailor virtual humans’ design to enhance their acceptance and increase their counseling effectiveness.

Virtual health assistants, digital characters designed to guide patients through healthcare interventions, offer scalable solutions, but their effectiveness may depend on user perceptions of realism. This study investigated how the visual fidelity and voice modality of virtual health assistants influence perceived realism, perceived voice naturalness, and intentions to screen for colorectal cancer among Black American adults aged 45–75. In a 2 × 4 factorial between-participants experiment (N = 266), participants were randomized to virtual health assistants varying in visual fidelity and voice modality. Structural equation modeling revealed that voice naturalness had a strong positive effect on perceived realism, which in turn significantly predicted screening intentions. Visual fidelity also contributed directly to perceived realism, although to a lesser extent. These results suggest that enhancing voice naturalness in virtual health assistants may be a critical design priority for improving engagement and promoting preventive health behaviors in digital health interventions.

Designing virtual humans to support mental health scenarios is challenging, as it requires designers to think of characteristics and strategies to promote rapport, increase trust, and favor the agents’ effectiveness during the intended interactions. This paper explores the impact of featuring the designer as a product cue in virtual human interactions, with a particular focus on how gender synchrony among users, virtual humans, and their designers influences trust and intentions towards well-being practices like gratitude journaling. We conducted an online, asynchronous, between-subjects user study and analyzed responses from 159 adult participants from the United States: Participants were female (n=80) or male (n=79). We assessed how gender synchrony could affect users’ intentions and the perceived trustworthiness of the virtual agents. Our findings suggest that gender synchrony does not significantly impact users’ intentions towards gratitude journaling. However, significant differences were observed concerning source credibility linked to agent trustworthiness when there was an age gap of less than ten years between users and virtual humans. These findings underscore the importance of considering both gender and age when designing virtual humans, as these factors can significantly enhance user acceptance and could foster the overall effectiveness of virtual human interactions in sensitive contexts such as mental health. Our study contributes to understanding how demographic characteristics and the use of the designer as a product cue can be strategically adapted to improve user experience in mental health scenarios.

Control over an avatar in virtual reality can improve one's perceived sense of agency and embodiment towards their avatar. Yet, the relationship between control on agency and embodiment remains unclear. This work aims to investigate two main ideas: (1) the effectiveness of currently used metrics in measuring agency and embodiment and (2) the relationship between different levels of control on agency, embodiment, and cognitive performance. To do this, we conducted a between-participants user study with three conditions on agency ($\mathrm{n}=57$). Participants embodied an avatar with one of three types of control (i.e., Low - control over head only, Medium - control over head and torso, or High - control over head, torso, and arms) and completed a Stroop test. Our results indicate that the degree of control afforded to participants impacted their embodiment and cognitive performance but, as expected, could not be detected in the self-reported agency scores. Furthermore, our results elucidated further insights into the relationship between control and embodiment, suggesting potential uncanny valley-like effects. Future work should aim to refine agency measures to better capture the effect of differing levels of control and consider other methodologies to measure agency.

Virtual humans are employed in various contexts, including mental health interventions, to encourage users to adopt healthy behaviors. Establishing rapport with users can enhance the effectiveness of these virtual agents. One way to build rapport is by matching the personality between the virtual human and the participant, which may elicit a similarity-attraction effect, shown to increase trust and likeability in interactions. Despite the potential of computer-generated voices to convey personality through the manipulation of vocalic properties, prior research has primarily focused on non-vocal aspects of personality. To address this gap, we conducted an online study that altered a virtual human's vocalic properties to represent high or low extroversion, with a focus on the role of rapport in promoting mental wellness. In this study, a virtual human provided information on stress-reducing mental-wellness practices to 165 participants. Our findings suggest that synthesizing vocalic properties to resemble a low extroversion level can enhance the persuasiveness of virtual humans and improve rapport, as indicated by participants' self-reported intention to engage in mental-wellness practices.

Virtual humans (VHs) have the potential to support mental wellness among college computer science (CS) students. However, designing effective VHs for counseling purposes requires a clear understanding of students’ demographics, backgrounds, and expectations. To this end, we conducted two user studies with 216 CS students from a major university in North America. In the first study, we explored how students co-designed VHs to support mental wellness conversations and found that the VHs’ demographics, appearance, and voice closely resembled the characteristics of their designers. In the second study, we investigated how the interplay between the VH’s appearance and voice impacted the agent’s effectiveness in promoting CS students’ intentions toward gratitude journaling. Our findings suggest that the active participation of CS students in VH design leads to the creation of agents that closely resemble their designers. Moreover, we found that the interplay between the VH’s appearance and voice impacts the agent’s effectiveness in promoting CS students’ intentions toward mental wellness techniques.

Redirected walking can effectively utilize a user’s physical space when traversing larger virtual environments by using virtual self-motion gains for a user’s physical motions. In particular, curvature gain presents unique advantages in redirection but can lead to suboptimal orientations. To prevent this and add additional utility in redirected walking, we formally present strafing gain. Strafing gain seeks to add incremental lateral movements to a user’s position causing the user to walk along a diagonal trajectory while maintaining the original orientation of the user. In a study with 27 participants, we tested 11 values to determine the detection thresholds of strafing gain. The study, which was modeled on prior detection threshold studies, found that strafing gain could successfully redirect participants to walk along a 5.57° diagonal to the right and a 4.68° diagonal to the left. Furthermore, a supplementary study with 10 participants was conducted, verifying that orientation was maintained throughout redirection and validating the obtained detection thresholds. We discuss the implications of these findings and potential ways of improving these quantities in real-world applications.

Counseling requires intimacy between a counselor and a patient to reach healing and growth. However, building rapport between virtual human counselors and computing college students is a complex problem. It requires understanding students' experiences and goals, as also the effects the characteristics of a virtual human counselor, like the spoken accent, have in the interaction with a patient in regards to messenger credibility and self-disclosure. This paper reports findings of how virtual human counselors' spoken accents impact computing undergraduate students' mental wellness conversations in regard to students' self-reported multilingual skills: monolingual or multilingual. We developed two English-speaking rapport-building virtual humans, each with a different spoken English accent-American or German, to interview 62 North American undergraduate computing students from a North American campus. Our findings suggest that virtual humans' spoken accents impacted students' perceptions of the virtual humans' speaking skills. Additionally, we found a similarity-attraction effect between monolingual English speakers and the American-English-accented virtual human counselor concerning participants' engagement and perceptions of the virtual human's speaking skills.

Virtual human interactions are increasingly used for empathy skills training in healthcare by providing feedback during or after the interaction. The post-interview feedback consists of evaluation results of users' empathic responses and can be provided once without interfering with the interaction. However, this type of feedback is insufficient to engage trainees in obtaining a deeper understanding and insights into their learning. The scaffolded ping-pong feedback consists of a multi-round of descriptions explaining how to formulate desired empathic responses to induce users to explore an understanding of how to respond empathically. To increase the training effectiveness to enhance users' expressed empathy, we studied how to apply scaffolded ping-pong feedback in virtual human interactions to train users' empathy skills. In this paper, we studied how different forms of feedback impact users learning how to express empathy to screen-based virtual humans. To evaluate the training effectiveness, we collected 638 empathic responses from 27 clinician participants in the interaction with two virtual patients integrated with scaffolded ping-pong feedback. We compared them with 809 empathic responses from 25 clinician participants in the post-interview condition. The result shows that the scaffolded ping-pong feedback helped clinician participants to provide a higher percentage of medium-empathy level and a lower percentage of low-empathy level empathic responses than the post-interview feedback. The scaffolded ping-pong feedback was perceived as more difficult to use but did not affect the overall interaction experience with virtual patients. This work demonstrates the applicability of integrating ping-pong feedback to strengthen the training effectiveness of virtual human education interventions.

Virtual humans demonstrate the ability to act as non-judgmental conversational partners, eliciting greater self-disclosure. However, it is unclear what virtual human and conversational characteristics are important when self-disclosing. To address this gap, we conducted a set of qualitative, semi-formal interviews (n = 17) among computer science students to investigate participant mental models of willingness to disclose to virtual humans and characteristics of virtual humans that affect their self-disclosure. Our findings indicate that participants' mental models of virtual humans are largely inconsistent with current literature. This inconsistency appears to eliciting hesitancy and discomfort with virtual humans. Furthermore, trust and listening were identified as two primary characteristics of a virtual human interaction that are valuable towards willingness to disclose. Additionally, these characteristics were also valued in different ways for virtual humans in comparison to real humans. From the interviews, we identify and provide guidelines of designing virtual human interactions and conversations to elicit greater willingness to disclose.

Simulating real-world experiences in a safe environment has made virtual human medical simulations a common use case for research and interpersonal communication training. Despite the benefits virtual human medical simulations provide, previous work suggests that users struggle to notice when virtual humans make potentially life-threatening verbal communication mistakes inside virtual human medical simulations. In this work, we performed a 2x2 mixed design user study that had learners (n = 80) attempt to identify verbal communication mistakes made by a virtual human acting as a nurse in a virtual desktop environment. A virtual desktop environment was used instead of a head-mounted virtual reality environment due to Covid-19 limitations. The virtual desktop environment experience allowed us to explore how frequently learners identify verbal communication mistakes in virtual human medical simulations and how perceptions of credibility, reliability, and trustworthiness in the virtual human affect learner error recognition rates. We found that learners struggle to identify infrequent virtual human verbal communication mistakes. Additionally, learners with lower initial trustworthiness ratings are more likely to overlook potentially life-threatening mistakes, and virtual human mistakes temporarily lower learner credibility, reliability, and trustworthiness ratings of virtual humans. From these findings, we provide insights on improving virtual human medical simulation design. Developers can use these insights to design virtual simulations for error identification training using virtual humans.

In the context of mental wellness support, trust and intimacy between a counselor and a patient are necessary to converge healing processes positively. However, convincing students to trust a virtual human for topics regarding mental wellness is a complex problem that requires understanding students' experiences. Based on research that discusses mental health as a concerning topic regarding Computer Science (CS) students, this paper investigates how undergraduate computing-related students perceive virtual humans' expertise on mental wellness support based on demographic resemblance on spoken accent and gender. Four virtual human counselors were developed to conduct the study, as 58 undergraduate computing-related students from two North American universities were recruited and assessed. Our findings suggest that students were less inclined to interact with a male virtual human than a female one. Also, that spoken accents can impact students' perceptions of expertise under students' multilingualism.

This paper investigates how a virtual human's persuasion attempts influence the user's intentions to perform the recommended behaviors using the Theory of Planned Behavior. The Theory of Planned Behavior suggests that users' attitudes towards the behavior, subjective norms, and perceived behavior control determine user intentions to perform behaviors. Using the Theory of Planned Behavior, we identify the underlying mechanisms of how users' attitudes, subjective norms, and perceived behavior control influence the effectiveness of virtual human's persuasive attempts on user's intentions to perform the behavior. To identify the underlying mechanisms, we conducted an online study with 202 college students. In a between-subjects study, a virtual human persuaded students to use a mental health coping skill using six different persuasion strategies. We present evidence that persuasion strategies influenced the students' perceived behavior control, which further influenced the user intentions to perform the behavior. Additionally, the paper also shows that user personality influenced the effect of persuasion strategies on students' perceived behavior control. This knowledge of underlying mechanisms of how virtual human's persuasion attempts to influence users' intentions to perform the recommended behavior can help in designing effective intelligent virtual humans for persuasion.

This report investigates how clinician-participants react to virtual patients’ sensical vs. nonsensical responses in a training simulation that aims to help clinicians acquire empathetic skills toward high-risk patients with symptoms of the Suicide Crisis Syndrome (SCS). Two suicidal virtual patients were developed, and clinician-participants interactions with them were recorded. Their facial emotions were analysed in three key moments: after a baseline sensical response, after a nonsensical response, and after the following sensical response. We compared their basic facial emotions aggregated into Negative and Positive facial affective behaviors (FABs). We describe our study involving ten clinician-participants and the results of the facial expression analysis with Noldus FaceReader. Our results suggest that nonsensical responses from virtual humans have an overall impact in both Positive and Negative facial affective emotions, and may lead to an increased percent of time participants demonstrate Negative facial affective behaviors when interacting with virtual humans. We discuss several aspects regarding the impacts and importance of considering nonsensical responses in the context of virtual human-based interactions.

This paper investigates the process of automating the evaluation of empathetic response levels in virtual human interaction systems implementing mental health scenarios. Two suicidal virtual patients were developed to collect clinician participants' empathetic responses. Before collecting clinicians' responses, we tested the virtual human interaction with healthcare trainees. Trainees' empathetic responses were evaluated by experts to use the ECCS scale based on the ECCS level (Empathetic Communication Coding System). We trained classifiers using trainees' empathetic responses with experts' coded empathy levels as the training dataset. Clinician participants' empathetic responses to virtual patients were evaluated by experts and the classifiers. The performance of the classifiers was evaluated using the experts' coded level of clinicians' empathetic responses as a test dataset.

To influence user behaviors, Internet-based virtual humans (VH) have been used to deliver health interventions. However, Internet-based VH health interventions face challenges. The challenges can affect user perceptions of an Internet-based VH health intervention. In our work, we use an Internet-based VH health intervention to promote colorectal cancer (CRC) screening. We present design guidelines drawn from two studies. The two studies examined the influence of visual design and the influence of the information medium on user intentions to pursue more health information. In the first study, the analysis of the focus group (n=73 users) transcripts shows that the VH's visual realism, the VH's healthcare role, and the presence of a local healthcare provider's logo influenced user perceptions of the VH-based intervention's visual design. The findings from the focus groups were used to iterate the intervention and derive design guidelines. In the second study (n=1,400), the analysis of online surveys of users after the VH-based intervention showed that very few users focused on the VH's appearance. To influence the user intentions to pursue the health topic further, the results recommend the use of an animated VH to deliver health information compared to other mediums of information delivery, such as text. The design guidelines from the two studies can be used by developers to use VH-based interventions to influence users' intention to change behaviors.

Virtual humans are frequently used to help medical students practice communication skills. Here, we show that communication skills features drawn from the literature on best practices for doctor-patient communication can be used to predict student interviewers' success in a given domain skill. We also demonstrate the viability of Bayesian Rule Lists, an interpretable machine learning model, for this use case. Bayesian Rule Lists' predictive performance is comparable to that of other other commonly used algorithms, including decision trees. This suggests that Bayesian Rule Lists, which produce simple, human-readable trained binary classifiers, may be suitable for providing feedback for educational purposes.

Interaction with nature in virtual reality has been shown to induce similar restorative benefits as interaction with real-life nature. Drawing from Attention restoration theory, restorative benefits from being in virtual nature are likely to be improved through greater active engagement techniques with specific virtual natural features. Gamification is the process of adding game design elements in non-game scenarios in order to improve engagement and motivation. In the present pilot study, six participants completed either a gamified interaction with virtual nature, one where game design elements had been added in order to improve engagement with specific virtual nature features and thus possibly further facilitating sustained attention and self-reported restoration, having them pick plants and gain rewards such as a higher level in return, or a non-gamified task, one where they explored the virtual nature environment and looked at plants at their own pace without any game design elements. Gamified interaction improved sustained attention restoration more than non-gamified interaction. Additionally, gamified interaction was also shown to have reduced negative effect in self-reported restoration more than non-gamified interaction. While there are still several limitations, gamified interaction with virtual nature seems to offer vast potential as an engagement technique in improving sustained attention and self-reported restoration.

Because human-produced language is often still considered the golden standard in the natural language generation field, many virtual humans (VHs) used in communication-skills training simulations continue to be accompanied by human-authored dialogue instead of computer-generated dialogue. The human author used for writing such dialogue, however, has yet to be investigated. Dialogue authors can have a variety of identities - gender, culture, profession, etc. - which may or may not have an effect on the VH dialogue they produce. An identity incongruency between an author and the VH itself could have unexpected effects on VH interviewers' interactions. Therefore, in this paper, we examine the effect of an author's identity on VH dialogue produced in the context of a virtual patient (VP) interview. We evaluated writing samples from speech-language pathology (SLP) students and Chinese and Chinese Americans when creating a Chinese VP suffering from trouble swallowing. We also conducted a user study in which SLP students interviewed VPs created with dialogue from these different author identities. Our results indicate that there are some content differences in the dialogue produced by different author identities and that VH interviewers are able to recognize these differences.

Anonymous computer-based assessments are used to collect sensitive data in self-report assessments. However they lack human element for building rapport with users. Anonymous rapport building responsive virtual humans (VHs) are known to increase users' willingness to disclose honest information in self-report assessments. However, implementing responsive VHs is a complex task. It requires modeling verbal and nonverbal user behaviors in realtime and generating appropriate responses to address these behaviors. In our work, we investigate the use of non-responsive VHs as middle-ground between computers and responsive VHs to conduct self-report assessments. Prior research has shown that non-responsive VHs, which are unaware of users' behavior, can provide task-related support. We conducted a user study to evaluate the usability of non-responsive VHs to conduct self-report assessment. We also compared users' reported impression management and fear of negative evaluation with computer-based assessment to measure users' willingness to disclose honest information. We found that participants reported significantly higher usability for VH based assessment. Moreover, participants' willingness to disclose honest information with VH was same as in computer-based assessment. We conclude that non-responsive VHs can be used to improve user experience during time consuming self-report assessments.

Virtual humans are used to facilitate interactions in sensitive contexts such as healthcare. In such contexts, trust in the information source plays an important role in reception of the information. Prior work has shown that physical appearance affects trustworthiness in human-human interactions; therefore, we examined the effect of virtual human's appearance on users' trust. We ran a between-users study with 12 adult participants, who watched a video of a virtual human with professional attire (e.g., lab coat) or with general attire (e.g., button-down shirt). We examined the duration of eye fixation on the virtual human's face along with participants' self-reported trust levels. We found that there was no statistical difference in eye contact or trust between the two test conditions.

In the past few years, advances have been made on how mixed reality humans (MRHs) can be used for interpersonal communication skills training for medical teams; however, little research has looked at how MRHs can influence communication skills during training. One way to influence communication skills is to leverage MRHs as models of communication behavior. We created a mixed reality medical team training exercise designed to impact communication behaviors that are critical for patient safety. We recruited anesthesia residents to go through an operating room training exercise with MRHs to assess and influence residents' closed loop communication behaviors during medication administration. We manipulated the behavior of the MRHs to determine if the MRHs could influence the residents' closed loop communication behavior. Our results showed that residents' closed loop communications behaviors were influenced by MRHs. Additionally, we found there was a statistically significant difference between groups based on which MRH behavior residents observed. Because the MRHs significantly impacted how residents communicated in simulation, this work expands the boundaries for how VR can be used and demonstrates that MRHs could be used as tools to address complex communication dynamics in a team setting.

People often under or over-estimate their performance on interactive learning experiences with virtual agents, especially when it comes to interpersonal skills such as empathy. To generate more accurate self-assessment of performance, we propose the use of in-action reflective learning opportunities during interactive learning experiences with virtual agents. We conducted a user study in which third-year dental students (n=58) participated in an interactive learning experience that required them to demonstrate empathy towards a virtual agent playing the role of a patient. During the interaction, an in-action reflective learning intervention prompted the students to self-assess their performance with regards to empathy. Our results show that students' self-assessment correlates to an assessment performed by outside raters, and that students were significantly more empathetic on a second opportunity to demonstrate empathy to the virtual patient after having performed their reflection.

Virtual human interactions provide an important avenue for training as emergent opportunities arise. In response to a new training need, we propose a framework to rapidly create experiential learning opportunities in the form of a question--answer chat interaction with virtual humans. This framework takes quickly generated case documents and breaks down the case into small tasks that can be crowdsourced by nonexperts. This framework can serve as a first step to rapidly bootstrapping new virtual humans. We have applied our framework to the task of preparing health care students and professionals to infrequent, but high-stakes, situations such as infectious diseases, cranial nerve disorders, and stroke. Our framework was utilized by medical professionals interested in providing new training experiences to students and colleagues. Over the course of two months, these professionals created seven scenarios on a diverse range of topics that included Ebola, cancer, and neurological disorders. These scenarios were developed for multiple target audiences such as medical students, residents, and fellows. As a first step, each scenario utilized our framework and crowdsourced workers to create an initial corpus over the course of two days. From these seven cases, we selected two to evaluate the quality of the resulting virtual-human corpuses. The two scenarios were compared to preexisting reference scenarios that have been in curricular use for several years. We found a reduction in author time commitment of at least 92% while creating a character that was at least 75% as accurate as its reference counterparts. The commitment reduction and accuracy achieved by our framework represents a first step towards rapid development of a virtual human. Our framework can then be combined with other creation processes for further virtual-human development in order to create a mature virtual human. As part of a virtual-human development process, our framework can help to rapidly develop new scenarios in response to emergent training opportunities.

In a group setting, it is possible for attributes of one group member to indirectly affect how other group members are perceived. In this paper, we explore whether one group member's agency (e.g. if they are real or virtual) can indirectly affect behavior with other group members. We also consider whether variations in the agency of a group member directly affects behavior with that group member. To do so, we examined gaze behavior during a team training exercise, in which sixty-nine nurses worked with a surgeon and an anesthesiologist to prepare a simulated patient for surgery. The agency of the surgeon and the anesthesiologist were varied between conditions. Nurses' gaze behavior was coded using videos of their interactions. Agency was observed to directly affect behavior, such that participants spent more time gazing at virtual teammates than human teammates. However, participants continued to obey polite gaze norms with virtual teammates. In contrast, agency was not observed to indirectly affect gaze behavior. The presence of a second human did not affect participants' gaze behavior with virtual teammates.

In this paper we present a study exploring whether the physical presence of another human changes how people perceive and behave with virtual teammates. We conducted a study (n = 69) in which nurses worked with a simulated health care team to prepare a patient for surgery. The agency of participants' teammates was varied between conditions; participants either worked with a virtual surgeon and a virtual anesthesiologist, a human confederate playing a surgeon and a virtual anesthesiologist, or a virtual surgeon and a human confederate playing an anesthesiologist. While participants perceived the human confederates to have more social presence (p <; 0.01), participants did not preferentially agree with their human team members. We also observed an interaction effect between agency and behavioral realism. Participants experienced less social presence from the virtual anesthesiologist, whose behavior was less in line with participants' expectations, when a human surgeon was present.

This paper reports on applying constructionism with virtual agents in an educational setting. We introduce a methodology - Virtual Agent Constructionism, which involves health professions students creating virtual conversational agent representations of patients as part of coursework. The proposed methodology was implemented as an exercise in an educational course for three consecutive academic years. The aim of this paper is threefold - (i) to demonstrate feasibility of health professions students creating virtual agents in an educational setting as part of coursework, (ii) to report feedback from the students about the experience of creating virtual agents, and (iii) to report on initial trends that suggest that creating virtual agents helps health professions students improve their interviewing and interpersonal skills. In addition to these three innovations, we also present the virtual agents created as educational artifacts that can be used to train future students with their interpersonal skills.

A reflective practicum is a low-pressure, low-risk learning environment. In a reflective practicum a learner is educated in a professional practice and how to use reflection in the setting of that professional practice. An example of a low-pressure and low-risk learning environment is the use of embodied conversational agents (ECAs) in medicine to provide training for interviewing and diagnostic skills. However, such ECA experiences have not been used to teach how to use reflection in the setting of a professional practice. In this paper we present a framework that supports explicit reflective learning for ECA experiences. Using this framework, ECA experiences become a reflective practicum. This framework was applied to an ECA experience called the Neurological Examination Rehearsal Virtual Environment (NERVE), and created a sample experience called the NERVE Reflective Practicum (NERVE-RP). We conducted a user study in which second-year medical students (n = 76) used NERVE-RP and engaged in reflection based on the experience. The results of the user study show that students engage in valuable reflections during the experience including instances of critical reflection.

In this paper, we present the preliminary results of an ongoing study exploring how mixed-agency teams influence feelings of social presence. Participants worked with a team composed of either two virtual humans or a team composed of one virtual human and one real human. We found that while the presence of a human teammate did not affect overall feelings of social presence, the presence of a human teammate did appear to strengthen participants' perceptions that their virtual teammates were not real.

Welcome to the 21st IEEE Virtual Reality Conference, the premier international conference and exhibition on virtual reality!We are honored to host IEEE VR 2014 in Minneapolis, Minnesota, a vibrant center of culture, business, learning, and research in the upper midwest of the United States.The highlight of our outstanding technical program is the eighteen full-length research papers that are being published as a special issue of the IEEE Transactions on Visualization and Computer Graphics.We are indebted to the TVCG editor-in-chief, Ming Lin, for her vision and leadership in continuing to make this collaboration possible.Complementing these journal papers, our conference proceedings include ten short papers and fortyfour abstracts presenting an exciting collection of groundbreaking technical contributions.We thank this year's program chairs, Sabine Coquillart, Kiyoshi Kiyokawa, J. Edward Swan II, and Doug A. Bowman, for their dedicated leadership in soliciting papers, managing the multi-phase review process, and arranging the paper presentations at the conference.In addition to the long and short paper presentations, the week's technical events include the presentation of three panels, five workshops, three tutorials, thirty-one posters, thirteen research demonstrations, ten exhibits, and eight videos, bookended by keynote and capstone talks from VR visionaries Henry Fuchs and Hunter Hoffman.Accompanying these activities is the second annual Virtual Reality Doctoral Consortium, which with the generous support of the US National Science Foundation is providing a valuable mentoring opportunity to twelve promising PhD students working in the areas of virtual and augmented reality and 3D user interaction.We are further honored to again have the IEEE Symposium on 3D User Interfaces (3DUI) held jointly with the VR conference.The 9th installment of 3DUI features an outstanding program of papers, technotes, posters, and

We have created “You, M.D.”, an interactive museum exhibit in which users learn about topics in public health literacy while interacting with virtual humans. You, M.D. is equipped with a weight sensor, a height sensor and a Microsoft Kinect that gather basic user information. Conceptually, You, M.D. could use this user information to dynamically select the appearance of the virtual humans in the interaction attempting to improve learning outcomes and user perception for each particular user. For this concept to be possible, a better understanding of how different elements of the visual appearance of a virtual human affects user perceptions is required. In this paper, we present the results of an initial user study with a large sample size (n =333) ran using You, M.D. The study measured users’ reactions based on the user’s gender and body-mass index (BMI) when facing virtual humans with BMI either concordant or discordant from the user’s BMI. The results of the study indicate that concordance between the users’ BMI and the virtual human’s BMI affects male and female users differently. The results also show that female users rate virtual humans as more knowledgeable than male users rate the same virtual humans.

The 20th IEEE Virtual Reality Conference (VR 2013) held March 16th - 20th spans five days, with the first two days including five workshops and three tutorials. The following three days includes a strong program of papers, panels, posters, demos, and videos. Of special note is that this year featured both a high number of, and a higher than usual quality of, submissions. This results in a packed program that did require the panel sessions to coincide with a few paper sessions.

Health sciences students often practice and are evaluated on interview and exam skills by working with standardized patients (people that role play having a disease or condition). However, standardized patients do not exist for certain vulnerable populations such as children and the intellectually disabled. As a result, students receive little to no exposure to vulnerable populations before becoming working professionals. To address this problem and thereby increase exposure to vulnerable populations, we propose using virtual humans to simulate members of vulnerable populations. We created a mixed reality pediatric patient that allowed students to practice pediatric developmental exams. Practicing several exams is necessary for students to understand how to properly interact with and correctly assess a variety of children. Practice also increases a student's confidence in performing the exam. Effective practice requires students to treat the virtual child realistically. Treating the child realistically might be affected by how the student and virtual child physically interact, so we created two object interaction interfaces - a natural interface and a mouse-based interface. We tested the complete mixed reality exam and also compared the two object interaction interfaces in a within-subjects user study with 22 participants. Our results showed that the participants accepted the virtual child as a child and treated it realistically. Participants also preferred the natural interface, but the interface did not affect how realistically participants treated the virtual child.

Stressful interpersonal experiences can be difficult to prepare for. Virtual humans may be leveraged to allow learners to safely gain exposure to stressful interpersonal experiences. In this paper we present a between-subjects study exploring how the presence of a virtual human affected learners while practicing a stressful interpersonal experience. Twenty-six fourth-year medical students practiced performing a prostate exam on a prostate exam simulator. Participants in the experimental condition examined a simulator augmented with a virtual human. Other participants examined a standard unaugmented simulator. Participants reactions were assessed using self-reported, behavioral, and physiological metrics. Participants who examined the virtual human experienced significantly more stress, measured via skin conductance. Participants stress was correlated with previous experience performing real prostate exams; participants who had performed more real prostate exams were more likely to experience stress while examining the virtual human. Participants who examined the virtual human showed signs of greater engagement; non-stressed participants performed better prostate exams while stressed participants treated the virtual human more realistically. Results indicated that stress evoked by virtual humans is linked to similar previous real-world stressful experiences, implying that learners real-world experience must be taken into account when using virtual humans to prepare them for stressful interpersonal experiences.

Mixed reality (MR) combines virtual elements with the real, physical world. Virtual elements can be more dynamic - easily moving around or changing appearance. Physical elements are more static but improve immersion and presence. The combination of the two creates dynamic, immersive MR environments. However, individual entities in MR environments, e.g. an object or a character, are typically either purely physical or purely virtual. We propose using both virtual and physical elements in a hybrid virtual-physical entity. We implemented this idea with a hybrid virtual human and ran several user studies. These studies helped us develop guidelines for implementing hybrid entities.

The concepts of immersion and presence focus on the environment in a virtual environment. We instead focus on embodied conversational agents (ECAs). ECAs occupy the virtual environment as interactive partners. We propose that the ECA analogues of immersion and presence are physicality and social presence. We performed a study to determine the effect of an ECA's physicality on social presence and eliciting realistic behavior from the user. The results showed that increasing physicality can elicit realistic behavior and increase social presence but there was also an interaction effect with plausibility.

The articles in this special section contain selected papers from the IEEE Virtual Reality Conference (VR).

We investigate the efficacy of incorporating real-time feedback of user performance within mixed-reality environments (MREs) for training real-world tasks with tightly coupled cognitive and psychomotor components. This paper presents an approach to providing real-time evaluation and visual feedback of learner performance in an MRE for training clinical breast examination (CBE). In a user study of experienced and novice CBE practitioners (n = 69), novices receiving real-time feedback performed equivalently or better than more experienced practitioners in the completeness and correctness of the exam. A second user study (n = 8) followed novices through repeated practice of CBE in the MRE. Results indicate that skills improvement in the MRE transfers to the real-world task of CBE of human patients. This initial case study demonstrates the efficacy of MREs incorporating real-time feedback for training real-world cognitive-psychomotor tasks.

Welcome to IEEE Virtual Reality 2011! Similar to previous years, the conference program is mostly made up of single track presentations. There are fourteen long paper presentations, eighteen short presentations, and three panels. This year, we have some new improvements to the program including the addition of four invited presentations from the highly regraded IEEE Transactions on Visualization and Computer Graphics journal.

We present a new method for evaluating user experience in interactions with virtual humans (VHs). We code the conversational errors made by the VH. These errors, in addition to the duration of the interaction and the numbers of statements made by the participant and the VH, provide objective, quantitative data about the virtual social interaction. We applied this method to a set of previously collected interactions between medical students and VH patients and present preliminary results. The error metrics do not correlate with traditional measures of the quality of a virtual experience, e.g. presence and copresence questionnaires. The error metrics were significantly correlated with scores on the Maastricht Assessment of Simulated Patients (MaSP), a scenario-appropriate measure of simulation quality, suggesting further investigation is warranted.

THIS special section is dedicated to the three Best Papers of VRST 2009, which was held in Kyoto, Japan from 18 to 20 November 2009. The papers were first selected by a jury based on the referees’ reports and the oral presentation. The authors were then asked to provide an extended version. The extended papers were then reviewed through a strict peer review process. The first paper, “Robust Relocalization and Its Evaluation for Online Environment Map Construction” by Sehwan Kim, Christopher Coffin, and Tobias Hollerer from the University of California in Santa Barbara, presents robust methodology and evaluation for camera orientation relocalization, using virtual keyframes for online environment map construction. In case of tracking loss, incoming camera frames are matched against known-orientation keyframes to reestimate camera orientation. Instead of solely using real keyframes from incoming video, the proposed approach employs virtual keyframes which are distributed strategically within completed portions of an environment map. The authors compare different system variants using three evaluation methods to show that the proposed system is useful in a practical sense. To improve recovery robustness against lighting changes in indoor and outdoor environments, they propose a new approach based on illumination normalization and saturated area removal. In the second paper, “Natural Perspective Projections for Head-Mounted Displays,” Frank Steinicke, Gerd Bruder, Scott Kuhl, Pete Willemsen, Markus Lappe, and Klaus H. Hinrichs from Munster University in Germany analyze the user’s perception of a virtual environment displayed in a Head Mounted Displayed (HMD), which is rendered with different geometric fields of views. They introduce a psychophysical calibration method to determine the HMD’s actual field of view, which may vary from the nominal values specified by the manufacturer. Furthermore, we conducted two experiments to identify perspective projections for HMDs, which are identified as natural by subjects—even if these perspectives deviate from the perspectives that are inherently defined by the display field of view. In the third paper, “A Wide-View Parallax-Free EyeMark Recorder with a Hyperboloidal Half-Silvered Mirror and Appearance-Based Gaze Estimation,” Hiroki Mori, Erika Sumiya, Tomohiro Mashita, Kiyoshi Kiyokawa, and Haruo Takemura from Osaka University propose a wideview parallax-free eye-mark recorder with a hyperboloidal half-silvered mirror and a gaze estimation method suitable for the device. Their eye-mark recorder provides a wide field-of-view video recording of the user’s exact view by positioning the focal point of the mirror at the user’s viewpoint. They implemented and evaluated a gaze estimation method for their eye-mark recorder. They apply principal component analysis and multiple regression analysis to determine the relationship between the captured images and their corresponding gaze points.

Understanding the human-computer interface factors that influence users' behavior with virtual humans will enable more effective human-virtual human encounters. This paper presents evidence of a significant relationship between behavioral indicators of respect by users and virtual reality technology factors. Moreover, we found this evidence in an application domain where respect for others is fundamentally important, health professional education.

We integrated two simulators, a Virtual Patient (VP) system and a physical Human Patient Simulator (HPS) to create a mixed reality conscious sedation training environment. The VP system simulates clinical environments where learners conduct patient interviews. The HPS is a mannequin that models and simulates a wide array of clinical signs. The VP+HPS combination provides new capabilities specially suited for conscious sedation training. Future work will evaluate the efficacy of this MR simulation.

This paper proposes an approach to mixed environment training of manual tasks requiring concurrent use of psychomotor and cognitive skills. To train concurrent use of both skill sets, the learner is provided real-time generated, in-situ presented visual feedback of her performance. This feedback provides reinforcement and correction of psychomotor skills concurrently with guidance in developing cognitive models of the task.

Touch is a powerful component of human communication, yet has been largely absent in communication between humans and virtual humans (VHs). This paper expands on recent work which allowed unidirectional touch from human to VH, by evaluating bidirectional touch as a new channel for nonverbal communication. A VH augmented with a haptic interface is able to touch her interaction partner using a pseudo-haptic touch or an active-haptic touch from a co-located mechanical arm. Within the context of a simulated doctor-patient interaction, two user studies (n = 54) investigate how touch can be used by both human and VH to communicate. Results show that human-to-VH touch is used for the same communication purposes as human-to-human touch, and that VH-to-human touch (pseudo-haptic and active-haptic) allows the VH to communicate with its human interaction partner. The enhanced nonverbal communication provided by bidirectional touch has the potential to solve difficult problems in VH research, such as disambiguating user speech, enforcing social norms, and achieving rapport with VHs.

This paper proposes virtual social perspective-taking (VSP). In VSP, users are immersed in an experience of another person to aid in understanding the person's perspective. Users are immersed by 1) providing input to user senses from logs of the target person's senses, 2) instructing users to act and interact like the target, and 3) reminding users that they are playing the role of the target. These guidelines are applied to a scenario where taking the perspective of others is crucial - the medical interview. A pilot study (n = 16) using this scenario indicates VSP elicits reflection on the perspectives of others and changes behavior in future, similar social interactions. By encouraging reflection and change, VSP advances the state-of-the-art in training social interactions with virtual experiences.

This paper presents Mixed Reality Humans (MRHs), a new type of embodied agent enabling touch-driven communication. Affording touch between human and agent allows MRHs to simulate interpersonal scenarios in which touch is crucial. Two studies provide initial evaluation of user behavior with a MRH patient and the usability and acceptability of a MRH patient for practice and evaluation of medical students' clinical skills. In Study I (n=8) it was observed that students treated MRHs as social actors more than students in prior interactions with virtual human patients (n=27), and used interpersonal touch to comfort and reassure the MRH patient similarly to prior interactions with human patients (n=76). In the within-subjects Study II (n=11), medical students performed a clinical breast exam on each of a MRH and human patient. Participants performed equivalent exams with the MRH and human patients, demonstrating the usability of MRHs to evaluate students' exam skills. The acceptability of the MRH patient for practicing exam skills was high as students rated the experience as believable and educationally beneficial. Acceptability was improved from Study I to Study II due to an increase in the MRH's visual realism, demonstrating that visual realism is critical for simulation of specific interpersonal scenarios.

This paper proposes collocated After Action Review (AAR) of training experiences. Through Mixed Reality (MR), collocated AAR allows users to review past training experiences in situ with the user’s current, real-world experience. MR enables a user-controlled egocentric viewpoint, a visual overlay of virtual information, and playback of recorded training experiences collocated with the user’s current experience. Collocated AAR presents novel challenges for MR, such as collocating time, interactions, and visualizations of previous and current experiences. We created a collocated AAR system for anesthesia education, the Augmented Anesthesia Machine Visualization and Interactive Debriefing system (AAMVID). The system was evaluated in two studies by students (n=19) and educators (n=3). The results demonstrate how collocated AAR systems such as AAMVID can: (1) effectively direct student attention and interaction during AAR and (2) provide novel visualizations of aggregate student performance and insight into student understanding for educators.

This paper compares a large-screen display to a non-stereo head-mounted display (HMD) for a virtual human (VH) experience. As VH experiences are increasingly being applied to training, it is important to understand the effect of immersive displays on user interaction with VHs. Results are reported from a user study (n=27) of 10 minute human-VH interactions in a VH experience which allows medical students to practice communication skills with VH patients. Results showed that student self-ratings of empathy, a critical doctor-patient communication skill, were significantly higher in the HMD; however, when compared to observations of student behavior, students using the large-screen display were able to more accurately reflect on their use of empathy. More work is necessary to understand why the HMD inhibits students' ability to self-reflect on their use of empathy.

Virtual human (VH) experiences are receiving increased attention for training real-world interpersonal scenarios. Communication in interpersonal scenarios consists of not only speech and gestures, but also relies heavily on haptic interaction–interpersonal touch. By adding haptic interaction to VH experiences, the bandwidth of human-VH communication can be increased to approach that of human-human communication. To afford haptic interaction, a new species of embodied agent is proposed–mixed reality humans (MRHs). A MRH is a virtual human embodied by a tangible interface that shares the same registered space. The tangible interface affords the haptic interaction that is critical to effective simulation of interpersonal scenarios. We applied MRHs to simulate a virtual patient requiring a breast cancer screening (medical interview and physical exam). The design of the MRH patient is presented. This paper also presents the results of a pilot study in which eight (n = 8) physician-assistant students performed a clinical breast exam on the MRH patient. Results show that when afforded haptic interaction with a MRH patient, users demonstrated interpersonal touch and social engagement similarly to interacting with a human patient.

Mixed reality's (MR) ability to merge real and virtual spaces is applied to merging different knowledge types, such as abstract and concrete knowledge. To evaluate whether the merging of knowledge types can benefit learning, MR was applied to an interesting problem in anesthesia machine education. The virtual anesthesia machine (VAM) is an interactive, abstract 2D transparent reality simulation of the internal components and invisible gas flows of an anesthesia machine. It is widely used in anesthesia education. However when presented with an anesthesia machine, some students have difficulty transferring abstract VAM knowledge to the concrete real device. This paper presents the augmented anesthesia machine (AAM). The AAM applies a magic-lens approach to combine the VAM simulation and a real anesthesia machine. The AAM allows students to interact with the real anesthesia machine while visualizing how these interactions affect the internal components and invisible gas flows in the real world context. To evaluate the AAM's learning benefits, a user study was conducted. Twenty participants were divided into either the VAM (abstract only) or AAM (concrete+abstract) conditions. The results of the study show that MR can help users bridge their abstract and concrete knowledge, thereby improving their knowledge transfer into real world domains.

This paper proposes after-action review (AAR) with human-virtual human (H-VH) experiences. H-VH experiences are seeing increased use in training for real-world, H-H experiences. To improve training, the users of H-VH experiences need to review, evaluate, and get feedback on them. AAR enables users to review their H- VH interaction, evaluate their actions, and receive feedback on how to improve future real-world, H-H experiences. The Interpersonal Scenario Visualizer (IPSViz), an AAR tool for H-VH experiences, is presented. IPSViz allows medical students to review their interactions with VH patients. To enable review, IPSViz generates spatial, temporal, and social visualizations of H- VH interactions. Visualizations are generated by treating the interaction as a set of signals. Interaction signals are captured, logged, and processed to generate visualizations for review, evaluation and feedback. In a study (N=27), reviewing the visualizations helped students become self-aware of their actions with a virtual human and gain insight into how to improve interactions with real humans.

Any new tool introduced for education needs to be validated. We developed a virtual human experience called the Virtual Objective Structured Clinical Examination (VOSCE). In the VOSCE, a medical student examines a life-size virtual human who is presenting symptoms of an illness. The student is then graded on interview skills. As part of a medical school class requirement, thirty three second year medical students participated in a user study designed to determine the validity of the VOSCE for testing interview skills. In the study, participant performance in the VOSCE is compared to participant performance in the OSCE, an interview with a trained actor. There was a significant correlation (r(33)=.49, p<.005) between overall score in the VOSCE and overall score in the OSCE. This means that the interaction skills used with a virtual human translate to the interaction skills used with a real human. Comparing the experience of virtual human interaction to real human interaction is the critical validation step towards using virtual humans for interpersonal skills education.

This paper provides key insights into the construction and evaluation of interpersonal simulators--systems that enable interpersonal interaction with virtual humans. Using an interpersonal simulator, two studies were conducted that compare interactions with a virtual human to interactions with a similar real human. The specific interpersonal scenario employed was that of a medical interview. Medical students interacted with either a virtual human simulating appendicitis or a real human pretending to have the same symptoms. In Study I (n = 24), medical students elicited the same information from the virtual and real human, indicating that the content of the virtual and real interactions were similar. However, participants appeared less engaged and insincere with the virtual human. These behavioral differences likely stemmed from the virtual human's limited expressive behavior. Study II (n = 58) explored participant behavior using new measures. Nonverbal behavior appeared to communicate lower interest and a poorer attitude toward the virtual human. Some subjective measures of participant behavior yielded contradictory results, highlighting the need for objective, physically-based measures in future studies.

This paper reports on a study to examine the similarities and differences in experiencing an interpersonal scenario with real and virtual humans. A system that allows medical students to interview a life-size virtual patient using natural speech and gestures was used as a platform for this comparison. Study participants interviewed either a virtual patient or a standardized patient, an actor trained to represent a medical condition. Subtle yet substantial differences were found in the participants’ rapport with the patient and the flow of the conversation. The virtual patient’s limited expressiveness was a significant source of these differences. However, overall task performance was similar, as were perceptions of the educational value of the interaction.

For simulating hands-on tasks, the ease of enabling two-handed interaction with virtual objects gives Mixed Reality (MR) an expected advantage over Virtual Reality (VR). A user study examined whether two-handed interaction is critical for simulating hands-on tasks in MR. The study explored the effect of one- and two-handed interaction on task performance in a MR assembly task. When presented with a MR system, most users chose to interact with two hands. This choice was not affected by a user's past VR experience or the quantity and complexity of the real objects with which users interacted. Although two-handed interaction did not yield a significant performance improvement, two hands allowed subjects to perform the virtual assembly task similarly to the real-world task. Subjects using only one hand performed the task fundamentally differently, showing that affording two-handed interaction is critical for training systems.

A method is presented to rapidly incorporate real objects into virtual environments using laser scanned 3D models with color-based marker tracking. Both the real objects and their geometric models are put into a mixed environment (ME). In the ME, users can manipulate the scanned, articulated real objects, such as tools, parts, and physical correlates to complex computer-aided design (CAD) models. Our aim is to allow engineering teams to effectively conduct hands-on assembly design verification. This task would be simulated at a high degree of fidelity, and would benefit from the natural interaction afforded by a ME with many specific real objects.

University of Florida This paper presents a system which allows medical students to experience the interaction between a patient and a medical doctor using natural methods of interaction with a high level of immersion. We also present our experiences with a pilot group of medical and physician assistant students at various levels of training. They interacted with projector-based life-sized virtual characters using gestures and speech. We believe that natural interaction and a high level of immersion facilitates the education of communication skills. We present the system details as well as the participants ’ performance and opinions. The study confirmed that the level of immersion contributed significantly to the experience, and participants reported that the system is a powerful tool for teaching and training. Applying the system to formal communication skills evaluation and further scenario development will be the focus of future research and refinement.

We describe a between-subjects experiment that compared four different methods of travel and their effect on cognition and paths taken in an immersive virtual environment (IVE). Participants answered a set of questions based on Crook's condensation of Bloom's taxonomy that assessed their cognition of the IVE with respect to knowledge, understanding and application, and higher mental processes. Participants also drew a sketch map of the IVE and the objects within it. The users' sense of presence was measured using the Steed-Usoh-Slater Presence Questionnaire. The participants' position and head orientation were automatically logged during their exposure to the virtual environment. These logs were later used to create visualizations of the paths taken. Path analysis, such as exploring the overlaid path visualizations and dwell data information, revealed further differences among the travel techniques. Our results suggest that, for applications where problem solving and evaluation of information is important or where opportunity to train is minimal, then having a large tracked space so that the participant can walk around the virtual environment provides benefits over common virtual travel techniques.

We present algorithms that enable virtual objects to interact with and respond to virtual representations, avatars, of real objects. These techniques allow dynamic real objects, such as the user, tools, and parts, to be visually and physically incorporated into the virtual environment (VE). The system uses image-based object reconstruction and a volume query mechanism to detect collisions and to determine plausible collision responses between virtual objects and the avatars. This allows our system to provide the user natural interactions with the VE.We have begun a collaboration with NASA Langley Research Center to apply the hybrid environment system to a satellite payload assembly verification task. In an informal case study, NASA LaRC payload designers and engineers conducted common assembly tasks on payload models. The results suggest that hybrid environments could provide significant advantages for assembly verification and layout evaluation tasks.

Immersive virtual environments (VEs) provide participants with computer-generated environments filled with virtual objects to assist in learning, training, and practicing dangerous and/or expensive tasks. But for certain tasks, does having every object being virtual inhibit the interactivity? Further, does the virtual object's visual fidelity affect performance? Overall VE effectiveness may be reduced if users spend most of their time and cognitive capacity learning how to interact and adapting to interacting with a purely virtual environment. We investigated how handling real objects and how self-avatar visual fidelity affects performance on a spatial cognitive task in an immersive VE. We compared participants' performance on a block arrangement task in both a real-space environment and several virtual and hybrid environments. The results showed that manipulating real objects in a VE brings task performance closer to that of real space, compared to manipulating virtual objects.

No abstract available.

We present a system for generating real-time 3D reconstructions of the user and other real objects in an immersive virtual environment (IVE) for visualization and interaction. For example, when parts of the user's body are in his field of view, our system allows him to see a visually faithful graphical representation of himself, an avatar. In addition, the user can grab real objects, and then see and interact with those objects in the IVE. Our system bypasses an explicit 3D modeling stage, and does not use additional tracking sensors or prior object knowledge, nor do we generate dense 3D representations of objects using computer vision techniques. We use a set of outside-looking-in cameras and a novel visual hull technique that leverages the tremendous recent advances in graphics hardware performance and capabilities. We accelerate the visual hull computation by using projected textures to rapidly determine which volume samples lie within the visual hull. The samples are combined to form the object reconstruction from any given viewpoint. Our system produces results at interactive rates, and because it harnesses ever-improving graphics hardware, the rates and quality should continue to improve. We further examine realtime generated models as active participants in simulations (with lighting) in IVEs, and give results using synthetic and real data.