This paper presents a within-subject study to investigate the effects of leading and following behaviors on user visual attention behaviors when collaborating with a virtual agent (VA) during performing transportation tasks in immersive virtual environments. The task was to carry a target object from a location to a predefined location. There were two conditions, namely leader VA (LVA) and follower VA (FVA). The leader gave instructions to the follower to perform actions. In the FVA condition, users played the leader role, while they played the follower role in the LVA condition. The users and the VA communicated via spoken language. During the experiment, participants wore a head-mounted display and performed real walking in a room. In each condition, each participant performed 20 trials of object transportation for different types of objects. Our preliminary results revealed significant differences in the user visual attention behaviors between the follower and leader VA conditions during the transportation tasks.

We examined the effects on users during collaboration with two types of virtual human (VH) agents in object transportation in an immersive virtual environment or virtual reality (VR). The two types of virtual humans we examined are leader and follower agents. The goal of the users is to interact with the agents using natural language and carry objects from initial locations to destinations. In each trial, the follower agent follows a user’s instructions to perform actions to manipulate the object. The leader agent determines the appropriate actions that the agent and the user should perform. We developed a system which enabled users and virtual agents to carry objects in an intuitive manner. We conducted a within-subjects study to evaluate the user behaviors under two conditions: (LVH) interaction with a leader virtual human and (FVH) interaction with a follower virtual human. We found that the participants in LVH required a higher workload than that in FVH. However, the users’ experiences, game experiences, and users’ impressions between the two conditions were not significantly different.

This investigation compared the impact on the users’ non-verbal behaviors elicited by interacting with a crowd of emotional virtual humans (VHs) in native and non-native language settings. In a between-subject experiment, we collected objective metrics regarding the users’ behaviors during interaction with a crowd of VHs who expressed verbal and non-verbal communicative behaviors in response to users’ speech-based interaction. The study presented four VH crowd conditions based on their emotional disposition, namely Positive, Negative, Neutral, and Mixed, in accordance with which VHs exhibited the appropriate behaviors. Participants were tasked with collecting items in a virtual flea market, and they interacted using natural speech-based dialogue with the VHs. The language conditions were collected in the USA and under two different conditions in Taiwan. The participants in the USA group interacted with the VHs in English (a native language for the USA setting); and two different groups in Taiwan interacted with the VHs in either a foreign (English) or native (Mandarin) language, respectively. Results reveal that in the (TMA, TEN) and (UEN, TEN) conditions, native versus non-native language communication can alter social behaviors (e.g., interaction time) of participants towards virtual interlocutors. Our results also revealed strong effects of cultural differences on participants’ non-verbal social behaviors with VHs in the UEN and TMA conditions.

This investigation compared the emotional impact caused by a crowd of affective virtual humans (VHs) that communicated in the users' native or foreign language. We evaluated the users' affective reactions to a crowd of VHs that exhibited distinct emotional expressions. A total of four emotions were presented, which were Positive, Negative, Neutral, and Mixed. The VHs performed verbal and non-behaviors accordingly. Under the Mixed condition, the VHs were divided into three groups equally and each group was uniquely assigned one of the three emotions (i.e., positive, negative, and neutral). Users collected ten items from a virtual reality market. To complete the tasks, they interacted using natural speech with the emotional VHs. Three language conditions were investigated: one condition in USA and another two conditions in Taiwan. The group of participants in USA interacted with the VHs in English; and the two groups of participants in Taiwan interacted with the VHs using a foreign (English) language and a native (Mandarin) language respectively. We discovered that the medium of communication or language familiarity had a strong influence on participants' emotional reactions. When participants interacted in a foreign language with VHs with a positive emotional disposition, we found their positive emotional reactions were subdued and negative reactions were elevated. However, this was not the case when participants interacted with VHs in their native language, as their emotional reactions were contingent on the emotional disposition of the VHs.

In this contribution we examined the effects on users during interaction with a virtual human crowd in an immersive virtual reality environment. We developed an agent-based crowd model with rich properties including eye gaze, facial expression, body motion, and verbal and non-verbal behaviors. The scenario was a virtual market in which the users needed to gather specific items. In a betweensubjects design, users interacted with a virtual human crowd that showed opposite valenced emotional expressions. There are four conditions in the between-subjects design. These includes different affective virtual crowds namely, Positive, Negative, Neutral, and a Mix condition. The Mix group is defined by a combination of Positive, Negative and Neutral emotional expressive characters. Depending on the specific condition, the virtual humans showed specific verbal and non-verbal behaviors. During the experiment we collected objective measures such as skin electrodermal activity, total time in the simulation, the number of interactions with the agents and performance measures. The subjective measures were the differential emotional survey (DES), and a user experience survey. We reported our findings with an in-depth analysis.

Abstract This paper studies the user collaboration experience with proactive and reactive agents in transporting boxes in virtual environments. Two main characters, the avatar and the agent, are controlled by a user and a controller, respectively. The user and the agent communicate with each other by voice. The agent can be proactive or reactive. The user follows the instruction issued by the proactive agent, whereas the user instructs the reactive agent to perform actions. The goal is to transport boxes to goal positions with orientation constraints. We conducted a user study to analyze the behaviors of participants in several aspects, including task completion time, path length, control experience, and co‐presence experience. We report our findings and make suggestions for future development.