Deaf students face a persistent visual attention split between signer and instructional materials. Although virtual reality (VR) is often promoted as an educational solution, it typically reinforces hearing norms (e.g., caption overlays or interpreter boxes onto hearing classrooms). Our work foregrounds Deaf leadership and reclaims VR design authority: in a mixed-hearing team led by Deaf scholars, we designed and evaluated a VR classroom prototype featuring three signer-placement modes: corner, parallel, and transparent. Twelve Deaf participants explored the prototype during a 15-minute lecture and participated in qualitative semi-structured interviews. Participants reported reduced attention split and improved visibility, and suggested VR may support flexibility and comprehension in Deaf learning. From these reflections, we introduce a five-dimension conceptual framework—proximity, customizability, visual efficiency, cultural fit, and task flexibility—that organizes how Deaf signers evaluate signer placements. This work moves Deaf Tech theory into practice, opening pathways for future Deaf-centered, culturally grounded HCI.

Gesture interaction is standard in modern extended reality head-mounted displays, e.g., Microsoft HoloLens 2 and Apple Vision Pro. These gestures come from multiple sources, such as elicitation studies, the application designer, or an expert. However, it is unclear if the source of the gesture affects the memorability of gestures (i.e., level of recall). In a user study, 35 participants used the HoloLens 2 to test three gestures set from different sources: 1) expert-defined, whereby we asked four experts to define gestures; 2) user-defined, whereby each participant defined their gestures; and 3) elicitation-defined, whereby we used gestures proposed by previous work. Our results showed that expert-defined gestures (72.73% recall) were more likely to be remembered than elicitation-defined (59.87% recall) gestures. Further, user-defined gestures (88.75% recall) were the most memorable. These data suggest that the source of gestures has a substantial impact on their memorability.

In 3D environments, designing efficient notifications is crucial for capturing user attention. While visual notifications -such as objectattached and fixed-position ones- are commonly used in virtual environments, they often require users to shift their gaze away from task-relevant areas, which can interrupt workflow and delay responses. To address these limitations, we designed two gaze-based notification techniques to provide responsive and intuitive notification in a virtual reality (VR) cooking task. We evaluated four different notification types: two world-fixed notifications (onObject and onDock) and two gaze-based methods (GazeCue and Gaze+Dock) with 16 participants. Our results show that participants performed better in using gaze-based notifications compared to world-fixed ones. Questionnaire results indicated higher usability, greater perceived presence, and lower cognitive load for gaze-based notifications. These results highlight the effectiveness of gaze-based notification techniques in VR by reducing the need for visual search and minimizing cognitive load. Our findings provide insights for developers or engineers to design more intuitive and responsive user-interfaces for 3D environments.

Virtual reality (VR) delivers immersive and engaging experiences, yet accessibility remains a significant barrier for users with disabilities. Although prior research has identified key challenges and proposed solutions, it is unclear whether the advances have been adopted in real-world applications. We systematically evaluated current accessibility in Meta Quest 3 applications, analyzing interaction modes and support for mobility, auditory, visual, and cognitive impairments. Our findings reveal significant gaps: many apps lack critical accessibility features, particularly for vision, fine motor control, and audio accessibility, which are absent or minimally supported. This inconsistency suggests that accessibility is often treated as an optional enhancement rather than a core requirement. These results underscore the urgent need for standardized accessibility guidelines and inclusive design practices in VR development.

Visually searching for objects is an everyday task. In many contexts, people must visually search for multiple objects at the same time while avoiding distractor objects, such as triage during a mass casualty incident. While many prior augmented reality (AR) and virtual reality (VR) studies have investigated cues to aid in visual search tasks, few have investigated cues in contexts involving multiple targets and distractors with a full 360° effective field of regard (EFOR). Individually, multiple targets, distractors, and a full 360° EFOR each add complexity to visual search; when combined, they compound the difficulty even further. In this paper, we present such a study that compares three common types of visual cues (2D Wedge, 3D Arrow, and Gaze Line) to a baseline condition with no cueing for a 360° visual search task. Our results reinforce the importance of providing some type of cue, with the Gaze Line design being particularly beneficial. We discuss the potential implications of these findings for designing cues specifically for such complex visual search tasks.

Notifications are a fundamental aspect of daily computing, whether on desktops, laptops, smartphones, or smartwatches. On average, adults receive around 200 notifications per day—approximately one every five minutes during waking hours. As Extended Reality (XR) headsets advance, they may become the primary medium for digital interactions, making notification management a crucial factor in their usability. While notifications are known to be disruptive on smartphones, their impact could be even more pronounced on head-worn devices. To investigate this, we conducted an exploratory five-day study with eight participants wearing display-equipped smartglasses that delivered notifications from their smartphones. Participants used the glasses throughout their day for at least 2 hours receiving on average 62% of all notifications on the glasses, submitted daily journal entries, and participated in post-study interviews. We also logged notification sources and timestamps throughout the study. Our findings reveal both practical advantages and significant challenges of head-worn notification delivery. While participants appreciated the convenience and immediacy of glanceable alerts, concerns about privacy, social acceptability, and distraction emerged as key barriers to adoption.

Virtual reality (VR) forest bathing for stress relief and mental health has recently become a popular research topic. As people spend more of their lives indoors and have less access to the restorative benefit of nature, having a VR nature supplement has the potential to improve quality of life. However, the optimal design of VR nature environments is an active area of investigation with many research questions to be explored. One major issue with VR is the difficulty of rendering high-fidelity assets in real time without causing cybersickness, or VR motion sickness, within the headset. Due to this limitation, we investigate if the realism of VR nature is critical for the restorative effects by comparing a low-realism nature environment to a high-realism nature environment. We only found a significant difference in the perceived restorativeness of the two environments, but after observing trends in our data toward the stress reduction potential of the high-realism environment, we suggest exploring more varieties of high and low-realism environments in future work to investigate the full potential of VR and how people respond.

Virtual reality (VR) could support the need for easily accessible therapeutic techniques, such as viewing art and immersing oneself in nature. Our study searches for the optimal virtual environment (VE) by exploring whether beauty in moving and still VEs contributes to stress reduction and perceived restorativeness. We hypothesized that the moving forest environment would result in the most stress reduction, while the abstract art would result in the least, with additional comparisons to a still forest environment and a control condition. The control condition took place outside the virtual headset to simulate what stress reduction would look like without a nature intervention. After working with 78 participants, we found an increase in statistical significance for stress reduction and perceived restorativeness in the moving forest condition compared to the control, as measured by the Zuckerman Inventory of Personal Reactions (ZIPERS) positive affect and the Perceived Restorativeness Scale (PRS). Additionally, the PRS and heart rate measures showed greater restorativeness in the moving forest condition than in the abstract art condition. Heart rate measures also showed statistical significance between the forest image condition and the control and moving forest conditions.

Virtual Reality (VR) applications like OpenBrush offer artists access to 3D sketching tools within the digital 3D virtual space. These 3D sketching tools allow users to “paint” using virtual digital strokes that emulate real-world mark-making. Yet, users paint these strokes through (unimodal) VR controllers. Given that sketching in VR is a relatively nascent field, this paper investigates ways to expand our understanding of sketching in virtual space, taking full advantage of what an immersive digital canvas offers. Through a study conducted with the participation of artists, we identify potential methods for natural multimodal and unimodal interaction techniques in 3D sketching. These methods demonstrate ways to incrementally improve existing interaction techniques and incorporate artistic feedback into the design.

Implementing visual and audio notifications on augmented reality devices is a crucial element of intuitive and easy-to-use interfaces. In this paper, we explored creating intuitive interfaces through visual and audio notifications. The study evaluated user performance and preference across three conditions: visual notifications in fixed positions, visual notifications above objects, and no visual notifications with monaural sounds. The users were tasked with cooking and serving customers in an open-source Augmented-Reality sandbox environment called ARtisan Bistro. The results indicated that visual notifications above objects combined with localized audio feedback were the most effective and preferred method by participants. The findings highlight the importance of strategic placement of visual and audio notifications in AR, providing insights for engineers and developers to design intuitive 3D user interfaces.

Previous studies on Eye-Hand Coordination Training (EHCT) focused on the comparison of user motor performance across different hardware with cross-sectional studies. In this paper, we compare user motor performance with an EHCT setup in Augmented Reality (AR), Virtual Reality (VR), and on a 2D touchscreen display in a longitudinal study. Through a ten-day user study, we thoroughly analyzed the motor performance of twenty participants with five task instructions focusing on speed, error rate, accuracy, precision, and none. As a novel evaluation criterion, we also analyzed the participants’ performance in terms of effective throughput. The results showed that each task instruction has a different effect on one or more psychomotor characteristics of the trainee, which highlights the importance of personalized training programs. Regarding different display technologies, the majority of participants could see more improvement in VR than in 2D or AR. We also identified that effective throughput is a good candidate for monitoring overall motor performance progress in EHCT systems.

While augmented reality (AR) headsets provide entirely new ways of seeing and interacting with data, traditional computing devices can play a symbiotic role when used in conjunction with AR as a hybrid user interface. A promising use case for this setup is situated analytics. AR can provide embedded views that are integrated with their physical referents, and a separate device such as a tablet can provide a familiar situated overview of the entire dataset being examined. While prior work has explored similar setups, we sought to understand how people perceive and make use of visualizations presented on both embedded visualizations (in AR) and situated visualizations (on a tablet) to achieve their own goals. To this end, we conducted an exploratory study using a scenario and task familiar to most: adjusting light levels in a smart home based on personal preference and energy usage. In a prototype that simulates AR in virtual reality, embedded visualizations are positioned next to lights distributed across an apartment, and situated visualizations are provided on a handheld tablet. We observed and interviewed 19 participants using the prototype. Participants were easily able to perform the task, though the extent the visualizations were used during the task varied, with some making decisions based on the data and others only on their own preferences. Our findings also suggest the two distinct roles that situated and embedded visualizations can have, and how this clear separation might improve user satisfaction and minimize attention-switching overheads in this hybrid user interface setup. We conclude by discussing the importance of considering the user's needs, goals, and the physical environment for designing and evaluating effective situated analytics applications.

Annotation in 3D user interfaces such as Augmented Reality (AR) and Virtual Reality (VR) is a challenging and promising area; however, there are not currently surveys reviewing these contributions. In order to provide a survey of annotations for Extended Reality (XR) environments, we conducted a structured literature review of papers that used annotation in their AR/VR systems from the period between 2001 and 2021. Our literature review process consists of several filtering steps which resulted in 103 XR publications with a focus on annotation. We classified these papers based on the display technologies, input devices, annotation types, target object under annotation, collaboration type, modalities, and collaborative technologies. A survey of annotation in XR is an invaluable resource for researchers and newcomers. Finally, we provide a database of the collected information for each reviewed paper. This information includes applications, the display technologies and its annotator, input devices, modalities, annotation types, interaction techniques, collaboration types, and tasks for each paper. This database provides a rapid access to collected data and gives users the ability to search or filter the required information. This survey provides a starting point for anyone interested in researching annotation in XR environments.

Despite knowing exactly what an object looks like, searching for it in a person's visual field is a time-consuming and error-prone experience. In Augmented Reality systems, new algorithms are proposed to speed up search time and reduce human errors. However, these algorithms might not always provide 100% accurate visual cues, which might affect users' perceived reliability of the algorithm and, thus, search performance. Here, we examined the detrimental effects of automation bias caused by imperfect cues presented in the Augmented Reality head-mounted display using the YOLOv5 machine learning model. 53 participants in the two groups received either 100% accurate visual cues or 88.9% accurate visual cues. Their performance was compared with the control condition, which did not include any additional cues. The results show how cueing may increase performance and shorten search times. The results also showed that performance with imperfect automation was much worse than perfect automation and that, consistent with automation bias, participants were frequently enticed by incorrect cues.

As we develop computing platforms for augmented reality (AR) head-mounted display (HMDs) technologies for social or workplace environments, understanding how users interact with notifications in immersive environments has become crucial. We researched effectiveness and user preferences of different interaction modalities for notifications, along with two types of notification display methods. In our study, participants were immersed in a simulated cooking environment using an AR-HMD, where they had to fulfill customer orders. During the cooking process, participants received notifications related to customer orders and ingredient updates. They were given three interaction modes for those notifications: voice commands, eye gaze and dwell, and hand gestures. To manage multiple notifications at once, we also researched two different notification list displays, one attached to the user’s hand and one in the world. Results indicate that participants preferred using their hands to interact with notifications and having the list of notifications attached to their hands. Voice and gaze interaction was perceived as having lower usability than touch.

While the visual search for targets in a complex scene might benefit from using augmented-reality (AR) head-mounted display (HMD) technologies by helping to efficiently direct human attention, imperfectly reliable automation support could manifest in occasional errors. The current study examined the effectiveness of different HMD cues that might support visual search performance and their respective consequences following automation errors. A total of 56 participants searched a three-dimensional environment containing 48 objects in a room, in order to locate a target object that was viewed prior to each trial. They searched either unaided or assisted by one of the three HMD types of cues: an arrow pointing to the target, a plan-view minimap highlighting the target, and a constantly visible icon depicting the appearance of the target object. The cue was incorrect in 17% of the trials for one group of participants and 100% correct for the second group. Through both analysis and modeling of both search speed and accuracy, the results indicated that the arrow and minimap cues depicting location information were more effective than the icon cue depicting visual appearance, both overall, and when the cue was correct. However, there was a tradeoff on the infrequent occasions when the cue erred. The most effective AR-based cue led to a greater automation bias in which the cue was more often blindly followed without careful examination of the raw images. The results speak to the benefits of AR and the need to examine potential costs when AR-conveyed information may be incorrect because of imperfectly reliable systems.

Shinrin-yoku, also known as forest bathing, is a nature immersion practice that has been shown to have restorative effects on mental health. Recently, applications of shinrin-yoku in virtual reality (VR) have been investigated as means of providing similar mental health benefits to people that do not have direct access to nature. These applications have shown similar health benefits, although not to the extent of real nature. The factors that make VR nature immersion effective are little researched to date. This paper investigates the Biophilia Hypothesis in the context of a VR-based nature immersion experience. Twenty-six participants were immersed in a computer-generated virtual natural environment that was either high in biomass (forest) or devoid of biomass (canyon), after experiencing an arithmetic stressor task. We compared multiple restorative outcomes between the high and low biomass groups, as well as preference ratings for real and virtual high and low biomass scenes among all participants. Our results call for further investigation into data trends we observed.

As augmented reality (AR) technology and hardware become more mature and affordable, researchers have been exploring more intuitive and discoverable interaction techniques for immersive environments. This paper investigates multimodal interaction for 3D object manipulation in a multi-object AR environment. To identify the user-defined gestures, we conducted an elicitation study involving 24 participants and 22 referents using an augmented reality headset. It yielded 528 proposals and generated a winning gesture set with 25 gestures after binning and ranking all gesture proposals. We found that for the same task, the same gesture was preferred for both one and two-object manipulation, although both hands were used in the two-object scenario. We present the gestures and speech results, and the differences compared to similar studies in a single object AR environment. The study also explored the association between speech expressions and gesture stroke during object manipulation, which could improve the recognizer efficiency in augmented reality headsets.

Elicitation studies have become a popular method of participatory design. While traditionally used to examine unimodal gesture interactions, elicitation has started being used with other novel interaction modalities. Unfortunately, there has been no work that examines the impact of referent display on elicited interaction proposals. To address that concern this work provides a detailed comparison between two elicitation studies that were similar in design apart from the way that participants were prompted for interaction proposals (i.e., the referents). Based on this comparison the impact of referent display on speech and gesture interaction proposals are each discussed. The interaction proposals between these elicitation studies were not identical. Gesture proposals were the least impacted by referent display, showing high proposal similarity between the two works. Speech proposals were highly biased by text referents with proposals directly mirroring text-based referents an average of 69.36% of the time. In short, the way that referents are presented during elicitation studies can impact the resulting interaction proposals; however, the level of impact found is dependent on the modality of input elicited.

• All time information was reported as milliseconds but given as 100ths of a second.• Additionally, the time differences between gesture initiation and speech initiation were reported as being 60% of their actual value. This resulted in those time differences being reported as 6 times less than they were (e.g., 130ms should be 780ms).

In today's era of Internet of Things (IoT), where massive amounts of data are produced by IoT and other devices, edge computing has emerged as a prominent paradigm for low-latency data processing. However, applications may have diverse latency requirements: certain latency-sensitive processing operations may need to be performed at the edge, while delay-tolerant operations can be performed on the cloud, without occupying the potentially limited edge computing resources. To achieve that, we envision an environment where computing resources are distributed across edge and cloud offerings. In this paper, we present the design of CLEDGE (CLoud + EDGE), an information-centric hybrid cloud-edge framework, aiming to maximize the on-time completion of computational tasks offloaded by applications with diverse latency requirements. The design of CLEDGE is motivated by the networking challenges that mixed reality researchers face. Our evaluation demonstrates that CLEDGE can complete on-time more than 90% of offloaded tasks with modest overheads.

Individuals with Autism Spectrum Disorders (ASD) often display difficulty in social interaction, verbal and non-verbal communication skills. They often have problems with recognizing and interpreting gestures and mental states of others, which restricts their capacity to understand implicit information essential to the social awareness and, consequently, the performance of collaborative activities in face-to-face situations (co-located collaboration). This study aims to identify, using a systematic literature review, how co-located collaborative systems designed for individuals with ASD provide the users with collaboration support mechanisms to encourage the performance of collaborative activities and interaction with their partners. Twenty-six studies from six electronic databases between the years 2006 and 2018 were selected and analyzed. Results evidenced the inclusion of different strategies to enforce or stimulate the collaboration among users, although there is a lack of resources for collaboration support as well as of studies designed specifically for individuals with severe ASD.

This study explored interactions between users and a 3D virtual world using a multi-touch display and sought to identify the advantages of this form of interaction. An experiment was conducted with 28 subjects, who were classified into two groups: experienced gamers and casual gamers. The experiment involved a pseudo-universe, where users were required to find five objects using the multi-touch display, and five objects using a game controller (GamePad). Overall, multi-touch was slightly faster than the GamePad in the search for objects. Experienced users performed significantly better when using the GamePad, compared to casual gamers. When casual gamers were analyzed separately, their performance was better (at a statistically significant level) with the multi-touch display, compared to the GamePad. Finally, users had a significantly higher error rate when switching from the GamePad to the keyboard, than when switching from the multi-touch to the keyboard.

This research establishes a better understanding of the syntax choices in speech interactions and of how speech, gesture, and multimodal gesture and speech interactions are produced by users in unconstrained object manipulation environments using augmented reality. The work presents a multimodal elicitation study conducted with 24 participants. The canonical referents for translation, rotation, and scale were used along with some abstract referents (create, destroy, and select). In this study time windows for gesture and speech multimodal interactions are developed using the start and stop times of gestures and speech as well as the stoke times for gestures. While gestures commonly precede speech by 81 ms we find that the stroke of the gesture is commonly within 10 ms of the start of speech. Indicating that the information content of a gesture and its co-occurring speech are well aligned to each other. Lastly, the trends across the most common proposals for each modality are examined. Showing that the disagreement between proposals is often caused by a variation of hand posture or syntax. Allowing us to present aliasing recommendations to increase the percentage of users' natural interactions captured by future multimodal interactive systems.

The primary objective of this research is to understand how users manipulate virtual objects in augmented reality using multimodal interaction (gesture and speech) and unimodal interaction (gesture). Through this understanding, natural-feeling interactions can be designed for this technology. These findings are derived from an elicitation study employing Wizard of Oz design aimed at developing user-defined multimodal interaction sets for building tasks in 3D environments using optical see-through augmented reality headsets. The modalities tested are gesture and speech combined, gesture only, and speech only. The study was conducted with 24 participants. The canonical referents for translation, rotation, and scale were used along with some abstract referents (create, destroy, and select). A consensus set of gestures for interactions is provided. Findings include the types of gestures performed, the timing between co-occurring gestures and speech (130 milliseconds), perceived workload by modality (using NASA TLX), and design guidelines arising from this study. Multimodal interaction, in particular gesture and speech interactions for augmented reality headsets, are essential as this technology becomes the future of interactive computing. It is possible that in the near future, augmented reality glasses will become pervasive.

Recent years have seen an accelerated development of Virtual Reality (VR) and Augmented Reality (AR) systems. Through these systems, computers immerse users in a completely artificial environment or in one that is partially based on the user's physical surroundings, but partially created artificially. One primary objective of these systems is to cause the perception of actually “being there” (in the environment created by the computer). However, this perception of full immersion may be weakened by the need to use “unnatural” devices (e.g., game controller) to execute actions within the VR or AR environment. Accordingly, increased interest has emerged in developing mechanisms by which the user could interact with VR and AR environments using natural actions, such as the movements of his/her hands.

We present a whole-body gesture elicitation study using Head Mounted Displays, including a legacy bias reduction technique. The motivation for this study was to understand the type of gesture agreement rates for selection and manipulation interactions and to improve the user experience for whole-body interactions. We found that regardless of the production technique used to remove legacy bias, legacy bias was still found in some of the produced gestures.

We present a whole-body gesture elicitation study using Head Mounted Displays, including a legacy bias reduction. The motivation for this study was to understand the type of gesture agreement rates for selection and manipulation interactions and to improve the user experience for whole-body interactions. We looked at 23 participants and 20 distinct referents (with multiple gestures per referent). We found that regardless of the production technique used to remove legacy bias, legacy bias was still found in some of the produced gestures. In some instances, gestures were derived from previous interactions but were still appropriate for the environment presented. This study provides a rich set of information and useful recommendations for future designers and/or developers.

CubeVR incorporates the traditional studio-based learning in architecture with virtual reality (VR) and in doing so it enables features that are not available or difficult to achieve in the real-world. CubeVR allows students and instructors to move from the existing studio to a virtual environment, allowing for a more impactful learning experience. This paper presents our current state of the CubeVR prototype, informal impressions of users, and the use of different affordances in the virtual world.

Understanding gaming expertise is important in user studies. We present a study comprised of 60 participants playing a First Person Shooter Game (Counter-Strike: Global Offensive). This study provides results related to a keyboard model used to determine an objective measurement of gamers' skill. We also show that there is no correlation between frequency questionnaires and user skill.

CircGR is a multi-touch non-symbolic gesture recognition algorithm, which utilizes circular statistic measures to implement linearithmic (O(n log n)) template-based matching. CircGR provides a solution to gesture designers, which allows for building complex multi-touch gestures with high-confidence accuracy. We demonstrated the algorithm and described a user study with 60 subjects and over 12,000 gestures collected for an original gesture set of 36. The accuracy is over 99% with the Matthews correlation coefficient of 0.95. In addition, early gesture detection was successful in CircGR as well.

The complexity of game development has increased in the past 30 years, from a task that could almost be entirely handled by a single programmer to an endeavor requiring a large team. To reduce this complexity, we have developed a Domain-Specific Language (DSL) targeting the modeling of two-dimensional (2D) games. We call this language Eberos Game Modeling Language 2D (Eberos GML2D). By raising the level of abstraction through modeling, we allowed a simpler specification of the game, and reduced the time and programming efforts. In order to evaluate our approach, we modeled two games and compared the difference between the amount of work required to write the game from scratch and the amount required using our proposed language. These evaluations yielded promising results of 86.4% savings on programming effort, and 82.3% savings on programming time.

The use of domain-specific modeling languages (DSMLs) results in higher productivity during the development process. This is accomplished by raising the level of abstraction during design and focusing on domain concepts rather than low-level implementation details. Unlike other development paradigms, little work has been done in determining and measuring the claimed benefits of using DSMLs. In this paper, we propose a new approach to determine the effort involved in creating and using DSML models to develop applications and to manage the behavior of applications at runtime. The approach involves a classification of the effort involved, and definition of relevant metrics to measure the effort for each category. A case study is presented that shows how we applied the proposed metrics during the development and execution of an application using three different DSMLs. Categories and Subject Descriptors