As AR applications expand across our daily lives, understanding user interactions within mixed environments-where virtual and physical objects coexist-has become increasingly important. This work investigates human performance and behavior during visual search and selection tasks across three object conditions: (1) virtual objects only, (2) physical objects only, and (3) a combination of virtual and physical objects (Mixed) requiring frequent virtuality switching. We also vary the distance to the target plane while maintaining subtended visual angle: a ‘near’ condition at the headset's focal plane and a ‘far’ condition at a mid-zone action space distance of 3 meters. Results indicate that, while there are some small effects that can be linked back to established display phenomena such as Vergence-Accommodation Conflict, a main cause for performance differences among the object conditions comes from people adjusting their search and selection behavior to the challenges of virtuality switching, resulting in Mixed conditions requiring significant longer completion times, associated with significantly larger head motion, eye movement, and controller movement. Mixed conditions also resulted in significantly lower accuracy for target selection. Virtual-to-Physical transitions took the longest to complete, followed by Physical-to-Virtual transitions, both significantly longer than transitions to targets within the same virtuality. Participants also reported increased Eye Strain, Fatigue, and Task Load with the Mixed conditions. This work provides insight into the complexities of mixed object interaction and presents quantitative assessments of pronounced virtuality switching, with implications for designing effective AR interfaces.

Seamless integration of virtual and physical worlds in augmented reality benefits from the system semantically “understanding” the physical environment. AR research has long focused on the potential of context awareness, demonstrating novel capabilities that leverage the semantics in the 3D environment for various object-level interactions. Meanwhile, the computer vision community has made leaps in neural vision-language understanding to enhance environment perception for autonomous tasks. In this work, we introduce a multimodal 3D object representation that unifies both semantic and linguistic knowledge with the geometric representation, enabling user-guided machine learning involving physical objects. We first present a fast multimodal 3D reconstruction pipeline that brings linguistic understanding to AR by fusing CLIP vision-language features into the environment and object models. We then propose “in-situ” machine learning, which, in conjunction with the multimodal representation, enables new tools and interfaces for users to interact with physical spaces and objects in a spatially and linguistically meaningful manner. We demonstrate the usefulness of the proposed system through two real-world AR applications on Magic Leap 2: a) spatial search in physical environments with natural language and b) an intelligent inventory system that tracks object changes over time. We also make our full implementation and demo data available at (https://github.com/cy-xu/spatially_aware_AI) to encourage further exploration and research in spatially aware AI.

Sexual harassment has been recognized as a significant social issue. In recent years, the emergence of harassment in social virtual reality (VR) has become an important and urgent research topic. We employed a mixed-methods approach by conducting online surveys with VR users ( N = 166) and semi-structured interviews with social VR users ( N = 18) to investigate how users perceive sexual harassment in social VR, focusing on the influence of avatar appearance. Moreover, we derived users' response strategies to sexual harassment and gained insights on platform regulation. This study contributes to the research on sexual harassment in social VR by examining the moderating effect of avatar appearance on user perception of sexual harassment and uncovering the underlying reasons behind response strategies. Moreover, it presents novel prospects and challenges in platform design and regulation domains.

Dynamically adjusting the content of augmented reality (AR) applications to efficiently display information best fitting the available screen estate may be important for user performance and satisfaction. Currently, there is not a common practice for dynamically adjusting the content of AR applications based on their apparent size in the user's view of the surround environment. We present a Level-of-Detail AR mechanism to improve the usability of AR applications at any relative size. Our mechanism dynamically renders textual and interactable content based on its legibility, interactability, and viewability respectively. When tested, Level-of-Detail AR functioned as intended out-of-the-box on 44 of the 45 standard user interface Unity prefabs in Microsoft's Mixed Reality Tool Kit. We additionally evaluated impact on task performance, user distance, and subjective satisfaction through a mixed-design user study with 45 participants. Statistical analysis of our results revealed significant task-dependent differences in user performance between the modes. User satisfaction was consistently higher for the Level-of-Detail AR condition.

It is well documented that exergames are enjoyable to play and can significantly improve older adults’ health and well-being. However, there is limited research on exploring factors affecting these users’ acceptance of such games, especially in virtual reality (VR), a relatively newer technology. This study proposes an extended version of the Technology Acceptance Model (TAM). We use variables from TAM related to older Chinese adults and specific to VR exergames to explore and confirm critical factors that could influence these users’ acceptance of such games in VR. We tested the proposed model with 51 older Chinese adults (aged 65 and above) after playing three commercial VR exergames (Beat Saber, FitXR, Dance Central). Results show that these older adults who are younger and retired and have a higher education, better financial means, and a good health condition have a more positive view of VR exergames. In addition, Perceived Usefulness, Perceived Ease of Use, and Perceived Enjoyment positively affect the intention to play VR exergames. Self-Satisfaction has a positive impact on Perceived Ease of Use and Perceived Usefulness. However, unlike previous studies, our results suggest that Facilitating Conditions have a negative effect on Perceived Ease of Use. Finally, we discuss the theoretical and practical implications of our results.

Text selection is an essential activity in interactive systems, including virtual reality (VR) head-mounted displays (HMDs). It is useful for: sharing information across apps or platforms, highlighting and making notes while reading articles, and text editing tasks. Despite its usefulness, the space of text selection interaction is underexplored in VR HMDs. In this research, we performed a user study with 24 participants to investigate the performance and user preference of six text selection techniques (Controller+Dwell, Controller+Click, Head+Dwell, Head+Click, Hand+Dwell, Hand+Pinch). Results reveal that Head+Click is ranked first since it has excellent speedaccuracy performance (2nd fastest task completion speed with 3rd lowest total error rate), provides the best user experience, and produces a very low workload—followed by Controller+Click, which has the fastest speed and comparable experience with Head+Click, but much higher total error rate. Other methods can also be useful depending on the goals of the system or the users. As a first systematic evaluation of pointing $\times$ selection techniques for text selection in VR, the results of this work provide a strong foundation for further research in this area of growing importance to the future of VR to help it become a more ubiquitous and pervasive platform.

Uncertainty is widely acknowledged as an engaging gameplay element but rarely used in exergames. In this research, we explore the role of uncertainty in exergames and introduce three uncertain elements (false-attacks, misses, and critical hits) to an exergame. We conducted a study under two conditions (uncertain and certain), with two display types (virtual reality and large display) and across young and middle-aged adults to measure their effect on game performance, experience, and exertion. Results show that (1) our designed uncertain elements are instrumental in increasing exertion levels; (2) when playing a motion-based first-person perspective exergame, virtual reality can improve performance, while maintaining the same motion sickness level as a large display; and (3) exergames for middle-aged adults should be designed with age-related declines in mind, similar to designing for elderly adults. We also framed two design guidelines for exergames that have similar features to the game used in this research.

Mid-air hand interaction has long been proposed as a ‘natural’ input method for Augmented Reality (AR) systems. Current AR Head-Mounted Displays (HMDs) have a limited area for hand-based interactions. Because of this, users may easily move their hand(s) outside this tracked area during interaction, especially in dynamic tasks (e.g., when translating an object). Compared to common midair interaction issues, such as gesture recognition, arm/hand fatigue, and unnatural ways of interacting with virtual objects (e.g., selecting a distant object), boundary awareness issues in AR devices have received little attention. In this research, we explore visual techniques for boundary awareness in AR HMDs, focusing on object translation tasks. Through a systematic formative study, we first identify the challenges that users might face when interacting with AR HMDs without any boundary awareness information (i.e., how current systems work). Based on the findings, we then propose four methods (i.e., static surfaces, dynamic surface(s), static coordinated lines, and dynamic coordinate line(s)) and evaluate them against the benchmark (i.e., baseline condition without boundary awareness) to make users aware of the tracked interaction area. Our results show that visual methods for boundary awareness can help with dynamic mid-air hand interactions in AR HMDs, but their effectiveness and application are user-dependent.