Lightweight augmented reality (AR) glasses are increasingly entering everyday use, extending interaction design beyond short, isolated sessions. However, most existing gesture vocabularies are inherited from VR headsets or early AR goggles. These systems tend to prioritize recognizer accuracy while overlooking fatigue, sustainability, and social legibility in daily contexts. To address this gap, we collaborated with physical therapists (PTs) to reimagine gesture design for everyday AR, drawing on their expertise in safe and sustainable movement. Through a review of 104 AR applications, we identified 15 common gesture intents and implemented an on-device gesture generator. Ten licensed physical therapists, with an average of 14.8 years of professional experience, then shaped these gesture intents through three iterative stages: unaided gesture performance, PT-guided gesture substitution, and stage-aware card sorting. This work contributes (1) a PT-informed gesture translation method, (2) the Everyday-AR Golden Ergonomic Canvas, and (3) a stage-aware social legibility framework that illustrates how gesture suitability shifts with social readability. Together, these contributions provide a recognizer-agnostic reference framework for designing sustainable and socially coherent gesture vocabularies for lightweight AR glasses.

Augmented reality (AR) games hold promise for rehabilitation, yet most remain confined to laboratory studies with limited clinical uptake. Recent advances in spatial computing, especially lightweight, glasses-form-factor AR, create a timely opportunity to embed rehabilitative play into clinical practice and daily contexts. To investigate this potential, we systematically reviewed 132 applications and conducted playtesting with 14 licensed physical therapists. Our analysis revealed three ways therapists re-authored AR games: co-authored play (reshaping movements, progressions, and difficulty), situated play (adapting across specialties, conditions, and contexts), and dual play (mediating both physical recovery and psychological support). We reframe therapists’ frequent phrase—“It depends”—as a generative design principle. This study contributes a clinical reasoning–based framework and design principles and guidelines for creating personalized, situated forms of play that align with therapists’ everyday workflows and inform future lab-to-clinic translation.

Goal-directed throwing is ubiquitous in virtual reality (VR) games, demanding spatial and temporal accuracy to perceive position and motion, as well as to plan, and execute movement. Autistic individuals often show superior local visual sensitivity, which may paradoxically delay global visual processing, leading to incongruent visuomotor integration (VMI). Understanding VMI in VR and exploring supportive designs are vital for enhancing accessibility. We assessed VMI in VR with different levels of visual cluttering with autistic (n = 10) and non-autistic (n = 10) adults using overhand throwing with eye and hand tracking, comparing VR to identical physical environments and VR to visually simplified VR. In VR, all participants exhibited decreased throwing accuracy, increased visual scanning, and prolonged hand preparation phases. These differences were more significant in autistic individuals. However, our study highlighted an important effect: simplifying visual information in VR throwing resulted in autistic individuals outperforming non-autistic peers.

Difficulty in social interaction is a key factor in the diagnostic criteria for autism. Although not fully understood, fluid human social interaction demands a complex exchange of verbal and nonverbal signals, which is disrupted in autistic individuals. Differences in gaze behavior, gross motor movement, and physiological responses related to arousal and attention have been observed repeatedly in autistic individuals, potentially impacting social interaction. Our prior work (Simmons et al., 2023) uses a fully immersive virtual reality video game custom-designed to examine the role of social presence through solo and cooperative versions of the game. We predicted that the inclusion of a virtual presence would impact the temporal execution of gaze behaviors and gross motor movements, as well as modulate physiological arousal and blink rate in autistic individuals differently than nonautistic controls. We found that the cooperative condition produced a larger number of differences in gaze behaviors and gross motor movements for autistic individuals. In addition, arousal and blink rate displayed differences during the cooperative condition. These findings demonstrate the specific effects of a virtual social presence on fundamental behaviors that comprise social interaction and can be measured and eventually manipulated in virtual reality.

Autistic individuals commonly experience visuomotor integration (VMI) difficulties that impact upper extremity movements such as goal-directed throwing, which commonly exists in everyday activities and movement-based games. Autistic individuals often show superior ability towards visual details and inefficient global visual information processing, leading to incongruent VMI. Virtual reality (VR) provides adequate and accurate eye and hand movement data and allows readily individualized design, making it a good option for VMI assessment and potential game-based VMI intervention for autistic individuals. We conducted a pilot study with eight non-autistic young adults comparing visual-proprioceptive accuracy and VMI in VR and in an equivalent physical environment (PE) using a spatial estimation task and a goal-directed throwing task. We observed reduced visual and visual-proprioceptive accuracy in VR compared to PE. Success rates of throwing were lower in VR, with more gaze scanning of the environment and shorter fixation duration on task-relevant objects. Participants spent a longer time preparing for movement execution and had higher hand velocity at release in VR. Subsequent research will compare visual-proprioceptive accuracy and VMI between VR and PE in autistic individuals and explore their relationship with autistic traits.

Spatial awareness is the ability to perceive the spatial relationships among objects, including other people and one’s own body, in the surrounding environment. A strong sense of spatial awareness is an essential component of athletic performance, allowing players to move and react effectively in a fast-paced, dynamic environment. Characterizing spatial awareness with athletes on, for example, a soccer pitch presents major challenges, however. The eSports game, Rocket League, shares some feature of fast-paced, dynamic ball sports including an expectation of increased spatial awareness with increasing player skill. Because Rocket League is played on a screen, we sought to characterize player skill in terms of both objective game data and gaze behavior. We trained a YOLOv7 neural network to robustly identify the ball and players from each frame of gameplay to form areas of interest for gaze behavior characterization. We measured gameplay and gaze behavior from complete novices, players with some but limited gameplay exposure, and very experienced players. We see this pilot effort as a start to characterizing spatial awareness in Rocket League play as it develops with game skill, as well as an opportunity to examine how spatial awareness may transfer from the physical world to screen-based environments and vice-versa.

This paper introduces an integrated tool that combines aerobic exercise and second language learning (L2) for the purpose of enhancing adult cognitive health. The tool utilizes virtual reality (VR) technology to focus on auditory recognition and content comprehension aiming to improve language proficiency in potential adult L2 learners. The study investigates participants’ experiences, cognitive load demands, and usability of the integrated tool within a VR-based multitasking environment. Preliminary results indicate a positive trend in attitudes towards the tool’s usefulness, highlighting its potential for motivating exercise and learning, facilitating learning processes, and providing ease of use. We see a promising future in serious game that integrate physical exercise to motivate the uptake of health-promoting behaviors.

Fluid human social interaction demands a complex exchange of verbal and non-verbal signals. Effectively interpreting and responding to these signals requires timely visual attention to relevant stimuli and coordination of appropriate motor responses. Differences in social interaction are a key part of the criteria for autism, but such differences are also frequently found in conjunction with differences in gaze and gross motor behavior, as well as physiological arousal. This finding suggests these basic functions may contribute to atypical social interactions, especially in highly dynamic situations. Our study uses a virtual reality pattern completion video game with solo and cooperative conditions to measure gaze, motor, and physiological responses to social and non-social stimuli within the virtual environment. In a population of 11 autistic and 16 non-autistic young adults, we measured gaze behavior and pupil diameter as an index of physiological arousal and two different perception-action sequences that occur in the pattern completion task. Compared to non-autistic players, the autistic players had longer fixation duration times in cooperative condition during looks to the most socially dynamic region of the virtual environment. We also observed differences across group and condition in a perception action sequence that demanded visually localizing a block and placing it onto a specific location to complete the sequence. Pupil arousal responses were triggered on the same block placement event and revealed significantly larger arousal changes for autistic players in the cooperative condition. These findings demonstrate specific effects of social presence on fundamental behaviors that comprise social interaction and can be measured, and eventually manipulated, in virtual reality.

The Yucatan Peninsula contains many cenotes and underwater caves, some of which offer promising opportunities for archeological and paleontological research in the Americas. Here we describe the transdisciplinary research efforts as part of documenting and studying the skeletal deposits in the submerged Ice Age cave site of Hoyo N egro. We translate these efforts into a video game that represents the digital twin of Hoyo Negro for the public to explore. Through this game we seek to excite middle school children about opportunities in science through the rich content at Hoyo Negro and employing next generation science standards so that the game can be tested with middle school students and used as part of science curricula.

Gaze behavior is important in early development, and atypical gaze behavior is among the first symptoms of autism. Here we describe a system that quantitatively assesses gaze behavior using eye-tracking glasses. Objects in the subject’s field of view are detected using a deep learning model on the video captured by the glasses’ world-view camera, and a stationary frame of reference is estimated using the positions of the detected objects. The gaze positions relative to the new frame of reference are subjected to unsupervised clustering to obtain the time sequence of looks. The clustering method increases the accuracy of look detection on test videos compared against a previous algorithm, and is considerably more robust on videos with poor calibration.

Joint attention is an essential part of the development process of children, and impairments in joint attention are considered as one of the first symptoms of autism. In this paper, we develop a novel technique to characterize joint attention in real time, by studying the interaction of two human subjects with each other and with multiple objects present in the room. This is done by capturing the subjects' gaze through eye-tracking glasses and detecting their looks on predefined indicator objects. A deep learning network is trained and deployed to detect the objects in the field of vision of the subject by processing the video feed of the world view camera mounted on the eye-tracking glasses. The looking patterns of the subjects are determined and a real-time audio response is provided when a joint attention is detected, i.e., when their looks coincide. Our findings suggest a trade-off between the accuracy measure (Look Positive Predictive Value) and the latency of joint look detection for various system parameters. For more accurate joint look detection, the system has higher latency, and for faster detection, the detection accuracy goes down.