Figure 1: Experimental conditions: (1) Conversation Only (CON), (2) Conversation with Subtitles (CSU), (3) Conversation with Visualizations (CVI), and (4) Conversation with Subtitles and Visualizations (CSV).Each condition varied in the inclusion of textual and visual supports during the agent-led health conversation.

ABSTRACT This study examines how subtitles and image visualizations influence gaze behavior, working alliance, and behavior change intentions in virtual health conversations with ECAs. Visualizations refer to images on a 3D model TV and text on a virtual whiteboard, both reinforcing key content conveyed by the ECA. Using a 2 2 factorial design, participants were randomly assigned to one of four conditions: no subtitles or visualizations (Control), subtitles only (SUB), visualizations only (VIS), or both subtitles and visualizations (VISSUB). Structural equation path modeling showed that SUB and VIS individually reduced gaze toward the ECA, whereas VISSUB moderated this reduction, resulting in less gaze loss than the sum of either condition alone. Gaze behavior was positively associated with working alliance, and perceptions of enjoyment and appropriateness influenced engagement, which in turn predicted behavior change intentions. VIS was negatively associated with behavior change intentions, suggesting that excessive visual input may introduce cognitive trade‐offs.

ABSTRACT This study examines how multimodal communication strategies (subtitles, visualizations, and their combination), affect memory retention, attitudes toward behavior change, and cognitive load during long conversations (+20 min) in immersive virtual reality (VR). Using embodied conversational agents to educate participants on diabetes and healthy eating, we found that all conditions effectively improved memory retention and behavior change attitudes. However, the combination of multimodal strategies increased cognitive load, suggesting a trade‐off between engagement and cognitive demands. These findings highlight the potential of long VR conversations for healthcare education, while emphasizing the importance of balancing cognitive demands and exploring personalization for diverse users.

Research on interpersonal relationships has shown that individuals often exhibit different gaze patterns when viewing faces of other races compared to those similar to their own. In the context of Virtual Humans (VHs), participatory design processes frequently include features that reflect the designer's identity. This paper explores gaze behavior towards demographically diverse VHs in the context of emerging educational technology. Using Augmented Reality (AR), 33 participants engaged in a within-subjects study where their eye movements, learning outcomes, and perceptions were compared. Results indicate that gaze (no. of fixations and durations) was influenced by demographic concordance (when race and sex matched that of the VH). Additionally, learning outcomes positively correlated with the number of fixations, suggesting that increased visual focus may enhance learning. However, participants frequently misidentified the race of the Hispanic, Asian, and Black VH, revealing a potential perceptual misalignment likely shaped by implicit social categorization biases.

In virtual reality, creating intelligent virtual characters has been a long-lasting endeavor. However, while researchers have investigated several aspects of a virtual character’s intelligence, little attention has been paid to the impact of the implemented intelligence levels assigned to a virtual character during human–virtual character collaboration. Thus, we conducted a within-group study ( \(N=24\) ) to explore how three different intelligence levels (low vs. medium vs. high) assigned to a virtual character can impact how study participants perceive that virtual character and interact with the task they are instructed to complete. Specifically, for our study, we developed a jigsaw puzzle game and instructed our participants to solve it with the help of a virtual character. During the jigsaw puzzle solving process, we collected application logs related to how the participants executed the task and observed the virtual environment. Moreover, after each condition, we asked the participants to respond using a questionnaire that examined their social presence, how they perceived the character’s intelligence and compared it with their own, and how they rated the virtual character’s realism. Our results indicated that the different intelligence levels assigned to the virtual characters impacted participants’ responses on several variables, including co-presence, perceived intelligence, intelligence comparison, and character interaction and behavior realism. Moreover, based on the collected logged data, we found that the intelligence levels assigned to our virtual character significantly impacted the performance of our participants. Our results could be valuable to the research community for creating more engaging experiences with intelligent virtual characters for collaborative tasks in immersive environments.

Various experiments have led to a better understanding of how the surrounding environment influences long-term memory and spatial cognition, including those indicating how environmental context is encoded. However, specific external environment characteristics that may impact cognitive memory and performance remain unclear. Using an immersive Virtual Reality (VR) environment, we investigated the effects of color congruency on object placement accuracy, recall speed, and recognition.

Locomotion in virtual reality (VR) can be accomplished through various techniques. Some utilize controller-based interactions, while others use more advanced hardware. The current study focuses on systems that provide naturalistic movement. By building upon prior knowledge of previous omnidirectional system evaluations, this study compares users’ performance and preferences between four systems: Infinadeck, Cyberith Virtualizer Elite 2, EKTO VR Voyager Boots, and real-walking with teleportation.

Although researchers have explored how humans perceive the intelligence of virtual characters, few studies have focused on the ability of intelligent virtual characters to fix their mistakes. Thus, we explored the self-correction behavior of a virtual character with different intelligence capabilities in a within-group design ( \(N=23\) ) study. For this study, we developed a virtual character that can solve a jigsaw puzzle whose self-correction behavior is controlled by two parameters, namely, Intelligence and Accuracy of Self-correction . Then, we integrated the virtual character into our virtual reality experience and asked participants to co-solve a jigsaw puzzle. During the study, our participants were exposed to five experimental conditions resulting from combinations of the Intelligence and Accuracy of Self-correction parameters. In each condition, we asked our participants to respond to a survey examining their perceptions of the virtual character’s intelligence and awareness (private, public, and surroundings awareness) and user experiences, including trust, enjoyment, performance, frustration, and desire for future interaction. We also collected application logs, including participants’ dwell gaze data, completion times, and the number of puzzle pieces they placed to co-solve the jigsaw puzzle. The results of all the survey ratings and the completion time were statistically significant. Our results indicated that higher levels of Intelligence and Accuracy of Self-correction enhanced not only our participants’ perceptions of the virtual character’s intelligence, awareness (private, public, and surroundings), trustworthiness, and performance but also increased their enjoyment and desire for future interaction with the virtual character while reducing their frustration and completion time. Moreover, we found that as the Intelligence and Accuracy of Self-correction increased, participants had to place fewer puzzle pieces and needed less time to complete the jigsaw puzzle. Finally, regardless of the experimental condition to which we exposed our participants, they gazed at the virtual character for more time compared to the puzzle pieces and puzzle goal in the virtual environment.

This study investigated the extent that the non-verbal behaviors of virtual humans (VHs) and their socio-demographic attributes altered users' collision avoidance behaviors in Virtual Reality (VR). Users interacted with VHs representing different levels of ethnicities and gender, exhibiting different conditions of physical movement, and gaze behaviors. The VHs were depicted in three major ethnic conditions namely Asian, Caucasian, and Black. The physical movement states of the VHs were either static in the path of the user or walking toward the user in the opposite direction. The non-verbal gaze behavior of the VHs was either direct gaze or averted gaze. We used an HTC Vive tracking system to track users' performing real walking while we collected objective measures (i,e., continuous gaze, fixation gaze, clearance distance, and travel length), and subjective variables (i.e., game experiences and social presence). The results showed that the ethnicity of the VHs significantly impacted the gaze behavior of the users, and the gender of the VHs affected the user avoidance movement and their reciprocal gaze behavior. Our results revealed that users' physical movement, gaze behaviors, and collision avoidance were moderated by the VHs' perceived ethnicity, gender, and gaze behaviors. Understanding the impact of the socio-demographics attributes of VHs and their gaze behavior on users' collision avoidance is critical for applications in which users are navigating through virtual traffic, crowd, and other inter-personal simulations.

This paper presents a within-subject study to investigate the effects of leading and following behaviors on user visual attention behaviors when collaborating with a virtual agent (VA) during performing transportation tasks in immersive virtual environments. The task was to carry a target object from a location to a predefined location. There were two conditions, namely leader VA (LVA) and follower VA (FVA). The leader gave instructions to the follower to perform actions. In the FVA condition, users played the leader role, while they played the follower role in the LVA condition. The users and the VA communicated via spoken language. During the experiment, participants wore a head-mounted display and performed real walking in a room. In each condition, each participant performed 20 trials of object transportation for different types of objects. Our preliminary results revealed significant differences in the user visual attention behaviors between the follower and leader VA conditions during the transportation tasks.

We developed a virtual agent that motivates church-going users to change their health behavior by telling existing cultural narratives that have high relevance with the counseling topic in an engaging way. We evaluated this agent in a between-subjects experiment where participants interacted with an agent that counseled them on nutrition either without a story, with a story but told in a neutral speech style, or with a story using dramatic delivery inspired by church sermons. We found that interaction with either one of the storytelling agents leads to a significantly greater change in confidence to engage in the target behavior of healthy eating than interacting with a non-storytelling agent, demonstrating the efficacy of stories in health counseling by virtual agents.

Eye movement analysis in modern 3D rendering systems is reviewed and three new techniques are derived inspired by work developed in early Virtual Reality so-called 2.5D implementations, namely (a) gaze depth (i.e., vergence) estimation, (b) vergence calibration, and (c) real-time 3D event detection that considers eye- and head-coupling. The new 3D calibration shows excellent error reduction in terms of Mean Squared Error (MSE).

We examined the effects on users during collaboration with two types of virtual human (VH) agents in object transportation in an immersive virtual environment or virtual reality (VR). The two types of virtual humans we examined are leader and follower agents. The goal of the users is to interact with the agents using natural language and carry objects from initial locations to destinations. In each trial, the follower agent follows a user’s instructions to perform actions to manipulate the object. The leader agent determines the appropriate actions that the agent and the user should perform. We developed a system which enabled users and virtual agents to carry objects in an intuitive manner. We conducted a within-subjects study to evaluate the user behaviors under two conditions: (LVH) interaction with a leader virtual human and (FVH) interaction with a follower virtual human. We found that the participants in LVH required a higher workload than that in FVH. However, the users’ experiences, game experiences, and users’ impressions between the two conditions were not significantly different.

This investigation compared the impact on the users’ non-verbal behaviors elicited by interacting with a crowd of emotional virtual humans (VHs) in native and non-native language settings. In a between-subject experiment, we collected objective metrics regarding the users’ behaviors during interaction with a crowd of VHs who expressed verbal and non-verbal communicative behaviors in response to users’ speech-based interaction. The study presented four VH crowd conditions based on their emotional disposition, namely Positive, Negative, Neutral, and Mixed, in accordance with which VHs exhibited the appropriate behaviors. Participants were tasked with collecting items in a virtual flea market, and they interacted using natural speech-based dialogue with the VHs. The language conditions were collected in the USA and under two different conditions in Taiwan. The participants in the USA group interacted with the VHs in English (a native language for the USA setting); and two different groups in Taiwan interacted with the VHs in either a foreign (English) or native (Mandarin) language, respectively. Results reveal that in the (TMA, TEN) and (UEN, TEN) conditions, native versus non-native language communication can alter social behaviors (e.g., interaction time) of participants towards virtual interlocutors. Our results also revealed strong effects of cultural differences on participants’ non-verbal social behaviors with VHs in the UEN and TMA conditions.

This investigation compared the emotional impact caused by a crowd of affective virtual humans (VHs) that communicated in the users' native or foreign language. We evaluated the users' affective reactions to a crowd of VHs that exhibited distinct emotional expressions. A total of four emotions were presented, which were Positive, Negative, Neutral, and Mixed. The VHs performed verbal and non-behaviors accordingly. Under the Mixed condition, the VHs were divided into three groups equally and each group was uniquely assigned one of the three emotions (i.e., positive, negative, and neutral). Users collected ten items from a virtual reality market. To complete the tasks, they interacted using natural speech with the emotional VHs. Three language conditions were investigated: one condition in USA and another two conditions in Taiwan. The group of participants in USA interacted with the VHs in English; and the two groups of participants in Taiwan interacted with the VHs using a foreign (English) language and a native (Mandarin) language respectively. We discovered that the medium of communication or language familiarity had a strong influence on participants' emotional reactions. When participants interacted in a foreign language with VHs with a positive emotional disposition, we found their positive emotional reactions were subdued and negative reactions were elevated. However, this was not the case when participants interacted with VHs in their native language, as their emotional reactions were contingent on the emotional disposition of the VHs.

The commercialization and lowering costs of consumer grade Virtual Reality (VR) devices has made the technology increasingly accessible to users around the world. The usage of VR technology is often accompanied by an undesirable side effect called cybersickness. Cyber-sickness is the feeling of discomfort that occurs during VR experiences, producing symptoms similar to those of motion sickness. It continues to remain one of the biggest hurdles to the widespread adoption of VR, making it increasingly important to explore and understand the factors that influence its onset. In this work, we investigated the influence of the presence/absence of motion control on the onset and severity of cybersickness in an HMD based VR driving simulation employing steering as a travel metaphor. Towards this end, we conducted a between subjects study manipulating the presence of control between three experimental conditions, two of which (Driving condition and Yoked Pair condition) formed a yoked control design where every pair of drivers and their yoked pairs were exposed to identical vehicular motion stimuli created by participants in the driving condition. In the other condition (Autonomous Car condition), participants experienced a program driven autonomous vehicle simulation. Results indicated that participants in the Driving condition experienced higher levels of cybersickness than participants in the Yoked Pair condition. While these results don’t conform to findings from previous research which suggests that having control over motion reduces cybersickness, it seems to point towards the importance of the fidelity of the control metaphor’s feedback response in alleviating cybersickness. Simply allowing one control their motion may not readily alleviate cybersickness but could instead increase it in such HMD based VR driving simulations. It may hence be important to consider how well the control metaphor and its feedback matches users’ expectations if we want to successfully mitigate cybersickness.

The commercialization of Virtual Reality (VR) devices is making the technology increasingly accessible to users around the world. Despite the success that VR is starting to see with its growing popularity, it has yet to become widely adopted and achieve its ultimate goal- convincingly simulate real life like experiences. The inability to generate adequate levels of presence and to prevent the manifestation of cybersickness are the two prominent barriers that have hindered VR from achieving its ultimate goal. While traditional research has examined factors that influence (correlate with) the onset and severity cybersickness, there is still a gap in our knowledge about the consequences of having motion control on cybersickness in immersive virtual environments (IVE’s) achieved using tracked Head Mounted Displays (HMD’s). Furthermore, outside of a correlational capacity, it is still unclear as to what causes cybersickness to affect presence in immersive virtual environments. The success of immersive virtual reality as a technology will hence largely come down to our ability to understand the interrelationship between these variables and then address the challenges they pose. Towards this end, we investigated how the affordance of motion control affects cybersickness and presence in an HMD based VR driving simulation by conducting a between subjects study where we manipulated the affordance of control between three experimental conditions. We leverage structural equation modeling in an attempt to build a framework that explains the relationship between virtual motion control, workload, cybersickness, time spent in the simulation, perceived time and presence. Our structural model helps explain why motion control could be an important factor to consider in addressing VR’s challenges and realizing its ultimate aim to simulate reality.

In this contribution we examined the effects on users during interaction with a virtual human crowd in an immersive virtual reality environment. We developed an agent-based crowd model with rich properties including eye gaze, facial expression, body motion, and verbal and non-verbal behaviors. The scenario was a virtual market in which the users needed to gather specific items. In a betweensubjects design, users interacted with a virtual human crowd that showed opposite valenced emotional expressions. There are four conditions in the between-subjects design. These includes different affective virtual crowds namely, Positive, Negative, Neutral, and a Mix condition. The Mix group is defined by a combination of Positive, Negative and Neutral emotional expressive characters. Depending on the specific condition, the virtual humans showed specific verbal and non-verbal behaviors. During the experiment we collected objective measures such as skin electrodermal activity, total time in the simulation, the number of interactions with the agents and performance measures. The subjective measures were the differential emotional survey (DES), and a user experience survey. We reported our findings with an in-depth analysis.

We examined the effect of rendering style and the interplay between attention and emotion in users during interaction with a virtual patient in a medical training simulator. The virtual simulation was rendered representing a sample from the photo-realistic to the non-photorealistic continuum, namely Near-Realistic, Cartoon or Pencil-Shader. In a mixed design study, we collected 45 participants’ emotional responses and gaze behavior using surveys and an eye tracker while interacting with a virtual patient who was medically deteriorating over time. We used a cross-lagged panel analysis of attention and emotion to understand their reciprocal relationship over time. We also performed a mediation analysis to compare the extent to which the virtual agent’s appearance and his affective behavior impacted users’ emotional and attentional responses. Results showed the interplay between participants’ visual attention and emotion over time and also showed that attention was a stronger variable than emotion during the interaction with the virtual human.

In this contribution we studied how different rendering styles of a virtual human impacted users' visual attention in an interactive medical training simulator. In a mixed design experiment, 78 participants interacted with a virtual human representing a sample from the non-photorealistic (NPR) to the photorealistic (PR) rendering continuity. We presented five rendering style samples scenarios, namely low fidelity all Pencil Shaded (APS), Low to Mid Fidelity Pencil Shader on virtual patient (VP) only (PS), Mid Fidelity All Cartoon Shaded (ACT), Mid to High Fidelity Cartoon Shader on VP only (CT), and relatively High Fidelity Human Like (HL) appearance, and compared how visual attention differed between groups of users. For this study, we employed an eye tracking system for collecting and analyzing users' gaze during interaction with the virtual human in a failure to rescue medical training simulation. Results suggests that users may spend more time in the simulations on the non-realistic fidelity continuum that necessarily do not involve interaction with the virtual human. However, users preferred visually attending to virtual humans in the middle and high fidelity visual appearance conditions when engaging virtual humans in simulated social face-to-face dialogue as compared to the other conditions

We present an empirical evaluation on how stereoscopic viewing and haptic feedback deferentially affects fine motor perception-action coordination in a pick-and-place task in Virtual Reality (VR). The factors considered were stereoscopic viewing, haptic feedback, sensory-motor congruence and mismatch, and calibration on perception-action coordination in near field fine motor task performance in VR. Quantitative measures of placement error, distance, collision, and time to complete trials were recorded and analyzed. Overall, we found that participants' manual dexterous task performance was enhanced in the presence of both stereoscopic viewing and haptic feedback. However, we found that time to complete task was greatly enhanced by the presence of haptic feedback, and economy and efficiency of movement of the end effector as well as the manipulated object was enhanced by the presence of both haptic feedback and stereoscopic viewing. Whereas, number of collisions and placement accuracy were greatly enhanced by the presence of stereoscopic viewing in near-field fine motor perception-action coordination. Our research additionally shows that mismatch in sensory-motor stimuli can detrimentally affect the number of collisions, and efficiency of end effector and object movements in near-field fine motor activities, and can be further negatively affected by the absence of haptic feedback and stereoscopic viewing. In spite of reduced cue situations in VR, and the absence or presence of stereoscopic viewing and haptic feedback, we found that participants tend to calibrate or adapt their perception-action coordination rapidly with a set of at least 5 trials.

The commercialization of Virtual Reality (VR) devices has made it easier for everyday users to experience VR from the comfort of their living rooms. This recent uptake in VR has also increased reported incidents of cybersickness. Cybersickness refers to the discomfort experienced by an individual while experiencing virtual environments. The symptoms are similar to those of motion sickness but are more disorienting in nature resulting in dizziness, blurred vision, etc. Cybersickness is currently one of the biggest hurdles to the widespread adoption of VR, and it is therefore critical to explore the factors that influence its onset. Towards this end, we present a proof of concept simulation to study cybersickness in highly realistic immersive virtual environments.

Abstract This paper studies the user collaboration experience with proactive and reactive agents in transporting boxes in virtual environments. Two main characters, the avatar and the agent, are controlled by a user and a controller, respectively. The user and the agent communicate with each other by voice. The agent can be proactive or reactive. The user follows the instruction issued by the proactive agent, whereas the user instructs the reactive agent to perform actions. The goal is to transport boxes to goal positions with orientation constraints. We conducted a user study to analyze the behaviors of participants in several aspects, including task completion time, path length, control experience, and co‐presence experience. We report our findings and make suggestions for future development.

We describe a system for experiment standardization and distributing medication in medical trials using a virtual human. In our system, we employed a virtual experimenter that explains the experiment, medication, procedure and risks involved through a large screen display. The participant is able to interact with and ask questions of the virtual experimenter through a touch screen interface on an additional monitor display. During the interaction, the virtual experimenter will present the participant with a pill. The pill is physically distributed by a custom-made Arduino based dispenser. We conducted an initial user evaluation of the system using a placebo response protocol and a perceived pain scale. In the study, participants submerged their hand in a hot water bath before and after interacting with the system and reported their perceived pain response. The system was either a virtual human or text interface that either disseminated a pill stating that it was an analgesic or did not provide a medication. Through this system, we propose the potential use of virtual humans as a method to provide a consistent and standardized interaction between a participant and experimenter, while maintaining the benefits of social interaction in medication trials.

Creating realistic animations of virtual humans remains comparatively complex and expensive. This research explores the degree to which animation fidelity affects users' gaze behavior when interacting in virtual reality training simulations that include virtual humans. Participants were randomly assigned to one of three conditions, wherein the virtual patient either: 1) was not animated; 2) played idle animations; or 3) played idle animations, looked at the participant when speaking, and lip-synced speech and facial gestures when conversing with the participant. Each participant's gaze was recorded in an inter-personal interactive patient surveillance simulation. Results suggest that conversational and passive animations elicited visual attention in a similar manner, as compared to the no animation condition. Results also suggest that when participants face critical situations in inter-personal medical simulations, visual attention towards the virtual human decreases while gaze towards goal directed activities increases.

Realistic versus stylized depictions of virtual humans in simulated inter-personal situations and their ability to elicit emotional responses in users has been an open question for artists and researchers alike. We empirically evaluated the effects of near visually realistic vs. non-realistic stylized appearance of virtual humans on the emotional response of participants in a medical virtual reality system that was designed to educate users in recognizing the signs and symptoms of patient deterioration. In a between-subjects experiment protocol, participants interacted with one of three different appearances of a virtual patient, namely visually realistic, cartoon-shaded and charcoal-sketch like conditions in a mixed reality simulation. Emotional impact were measured via a combination of quantitative objective measures were gathered using skin Electrodermal Activity (EDA) sensors, and quantitative subjective measures such as the Differential Emotion Survey (DES IV), Positive and Negative Affect Schedule (PANAS), and Social Presence questionnaire. The emotional states of the participants were analyzed across four distinct time steps during which the medical condition of the virtual patient deteriorated (an emotionally stressful interaction), and were contrasted to a baseline affective state. Objective EDA results showed that in all three conditions, male participants exhibited greater levels of arousal as compared to female participants. We found that negative affect levels were significantly lower in the visually realistic condition, as compared to the stylized appearance conditions. Furthermore, in emotional dimensions of interest-excitement, surprise, anger, fear and guilt participants in all conditions responded similarly. However, in social emotional constructs of shyness, presence, perceived personality, and enjoyment-joy, we found that participants responded differently in the visually realistic condition as compared to the cartoon and sketch conditions. Our study suggests that virtual human appearance can affect not only critical emotional reactions in affective inter-personal training scenarios. but also users' perceptions of personality and social characteristic of the virtual interlocutors.