Intelligence and knowledgeability are sometimes treated interchangeably in virtual agents, yet they shape interaction in different ways. We disentangled these traits and tested how each drives human perceptions and interaction in virtual reality (VR). To address the lack of prior research examining both traits simultaneously, we created a VR application where participants collaborated with a virtual agent to complete a jigsaw puzzle while engaging in free-flowing conversation about the puzzle’s art piece. We manipulated intelligence through the virtual agent’s puzzle-solving ability and knowledgeability through its predefined depth of knowledge in art. Using a 2 × 2 within-group study, we collected perceptual responses, logged data, and qualitative feedback. Results showed intelligence significantly influenced perceptions of intelligence, knowledge, rapport, trust, co-presence, uncanny valley, and intelligence and knowledge comparisons, while knowledgeability impacted perceived knowledge, trust, and intelligence and knowledge comparisons. Interaction effects further highlighted their interdependence, offering design implications for virtual agents.

In this study, we examined the impact of agent familiarity and knowledgeability on several variables spanning agent perceptions (i.e., perceived knowledge, familiarity, trust, anthropomorphism, uncanny valley effect, and likability), social and emotional experiences (i.e., co-presence, rapport, cognitive process expectations, and willingness for future interaction), and conversation dynamics (i.e., conversation transcript, participants’ response word count, and response time). We created two virtual agents for the study: a digital replica of a professor from our department (i.e., familiar agent) and an agent with similar demographic variables (i.e., age, gender, and ethnicity) but with a fabricated appearance and voice (i.e., unfamiliar agent). We implemented both agents to exhibit two levels of knowledgeability (i.e., low and high) in the domain of game development and course-specific information. We used large language models (LLMs) to provide the agents with persona information and domain knowledge through prompt engineering. For our user study, we followed a 2 (familiarity: unfamiliar vs. familiar agent) \(\times\) 2 (knowledgeability: low vs. high knowledgeability) within-group study design and recruited 32 participants who engaged in a 5-minute, conversation-based virtual reality (VR) interaction with all four experimental conditions: unfamiliar agent with low knowledgeability (ULK), unfamiliar agent with high knowledgeability (UHK), familiar agent with low knowledgeability (FLK), and familiar agent with high knowledgeability (FHK). The findings demonstrated a significant main effect of agent familiarity on perceived knowledge, suggesting that familiarity plays a crucial role in shaping users’ perception of the agent’s knowledgeability level. Besides perceived knowledge, familiarity also affected all other variables, apart from co-presence. Conversely, agent knowledgeability affected perceived familiarity, trust, anthropomorphism, cognitive process expectations, willingness for future interaction, conversation content, and participants’ response word count. Finally, we found an interaction effect between agent familiarity and perceived knowledge, indicating that familiarity has a significant influence on users’ perceptions of the agent’s knowledgeability. This study contributes to the field of conversational human-agent interaction in VR by providing empirical evidence on how adapting both familiarity and knowledgeability of virtual agents can significantly enhance user experience, offering valuable insights into designing more engaging, trustworthy, and effective embodied conversational agents.

Motion tracking to project users into embodied virtual reality (VR) as avatars is an essential application of real-time computer graphics. Most current embodied VR systems rely on head-mounted displays (HMDs) to estimate user pose, as headset sensors can track the head and hands, thereby reconstructing the full body without the need for external hardware. However, measuring the quality of motion reconstruction algorithms from HMD-based tracking, particularly those intended for use in social settings, remains challenging due to the complex interaction between motion and perceived social signals. This paper compares two industrial tracking reconstruction solutions, called HMD1 (i.e., a basic HMD-based method that uses head tracking and hand positions estimated from HMD cameras) and HMD2 (i.e., an advanced HMD-based method with additional onboard camera streams), that estimate user motion using only an HMD against ground-truth motion capture (MoCap) data. It advocates for a social signal-based analysis that views motion as a communication medium and employs user observations to measure whether viewers successfully perceive the information encoded in motion. Across 156 socially expressive clips, Social Signal ratings were more effective than generic measures at revealing differences between the HMD methods. HMD2 preserved social meaning more accurately than HMD1, with fewer significant deviations from MoCap, while both HMD methods were frequently rated less natural than MoCap. A qualitative review localized recurrent failure modes, such as arm swivel/shoulder errors, posture reconstruction issues, and floating/stance artifacts, which help explain the misreading of social signals. We release a dashboard scorecard, motion capture data, and a benchmark protocol to enable consistent motion evaluation. More generally, this work advocates for an underexplored approach to motion evaluation that focuses on assessing the semantics of motion to determine quality. As reliance on generative artificial intelligence (AI) increases, it is essential to standardize evaluation to preserve the authenticity of the social signals conveyed. The developed dataset and the evaluation framework are provided on our project's website: https://github.com/facebookresearch/MotionIsTheMessageDataset.

We explored how responsiveness (i.e., the ability to answer questions) and awareness (i.e., the ability to navigate toward the user in the virtual environment) of a virtual agent acting as a tour guide impact study participants in a virtual museum. We followed a 2 (responsiveness: non-responsive vs. responsive virtual agent)$\times 2$(awareness: unaware vs. aware virtual agent) within-group ($N=29$) study design and conducted a study to explore several variables spanning: agent credibility and intelligence (i.e., perceived intelligence, perceived knowledge), social interaction and presence (i.e., co-presence, rapport), human-likeness (i.e., uncanny valley, anthropomorphism), awareness dimensions (i.e., private, public, and surrounding awareness), desire for future interaction, and behavioral responses (i.e., distance traveled, dwell gazes). We found that the responsive virtual agent positively impacted participants' perceived intelligence, perceived knowledgeability, co-presence, rapport, anthropomorphism, surrounding awareness, desire for future interaction, and dwell gaze on surroundings. However, the awareness factor did not impact our participants. Instead, we found responsiveness$\times$awareness interaction effects on distance traveled, dwell gaze on the virtual agent, and dwell gaze on surroundings. These findings offer valuable insights into designing intelligent virtual agents that act as museum tour guides, enhancing user experience in virtual museum settings.

We explored how study participants interacted with virtual characters in a virtual reality study. Specifically, we developed a 3 (proximity: close vs. middle vs. far) $\times 2$ (gesture: passive vs. active) experimental design (N = 26) to understand how combinations of proximity between two virtual characters and gestures assigned to them influence study participants’ self-reported ratings (co-presence, attentional allocation, behavioral interdependence, emotional reactivity, and perceived politeness). We also examined their avoidance movements (duration, trajectory length, and speed) and their avoidance decisions (passing through/around and minimum distance side). We collected both survey responses and our participants’ trajectories. Our study revealed that 1) the proximity factor impacted how our participants rated their co-presence and behavioral interdependence, as well as whether they decided to pass through or around the virtual characters, and 2) the gesture factor impacted how participants rated their behavioral interdependence, emotional reactivity, perceived politeness, and also affected their duration, trajectory length, and speed. Our research contributes to understanding personal space and social norms in virtual environments, offering valuable insights for virtual reality developers on the importance of social dynamics in designing interactions with virtual characters.

We explored the impact of depth of knowledge on conversational agents and human perceptions in a virtual reality (VR) environment. We designed experimental conditions with low, medium, and high depths of knowledge in the domain of game development and tested them among 27 game development students. We aimed to understand how the agent's predefined knowledge levels affected the participants' perceptions of the agent and its knowledge. Our findings showed that participants could distinguish between different knowledge levels of the virtual agent. Moreover, the agent's depth of knowledge significantly impacted participants' perceptions of intelligence, rapport, factuality, the uncanny valley effect, anthropomorphism, and willingness for future interaction. We also found strong correlations between perceived knowledge, perceived intelligence, factuality, and willingness for future interactions. We developed design guidelines for creating conversational agents from our data and observations. This study contributes to the human-agent interaction field in VR settings by providing empirical evidence on the importance of tailoring virtual agents' depth of knowledge to improve user experience, offering insights into designing more engaging and effective conversational agents.

Abstract We explored user interactions with a holographic sign language interpreter in a mixed reality (MR) classroom for deaf and hard of hearing students. The developed MR application projects a holographic signing avatar that translates in real time the lecture while a speaking instructor is teaching. Our study explored user interaction with the MR system, intending to provide design guidelines for digital MR sign language interpreters. We recruited eight participants and conducted a usability test focused on avatar framing (full‐body vs. half‐body) and avatar manipulation (fixed position, scale, and orientation vs. user‐adjustable position, scale, and orientation) in the MR classroom. We used a mixed‐method approach to analyze quantitative and qualitative data through recordings, surveys, and interviews. The results show user preferences toward viewing holographic signing avatars in the MR environment and user acceptability toward such applications.