Nonspeaking autistic individuals often build expressive communication through spelling on physical letterboards, but skilled one-on-one support is difficult to scale. XR letterboards could expand access, yet commodity near-hand press recognizers are brittle under the motor strategies common in nonspeaking typers (e.g., brief pokes, glides, folded-finger taps), leading to missed interactions that disrupt fluency and trust. We present SAGE (Spatial Anticipation of Gestural Events), a lightweight intent-inference layer that recovers intended selections even when no device press is registered. SAGE first fits a per-user, per-key spatial prior from historical device-registered taps, then fuses this prior at runtime with fingertip proximity and wrist-anchored pointing, using temporal peak detection to infer both press timing and letter identity. We evaluate SAGE on a HoloLens 2 AR letterboard dataset from four nonspeaking participants (244,620 hand-tracking samples) and compare against the headset recognizer and two physics-based replay baselines. On known taps, SAGE improves attribution quality (F1=0.91 vs. best baseline 0.52); on missed attempts, it recovers 71% of unregistered taps with correct letter identity for all recovered events. We contribute SAGE, the first dataset of nonspeaking XR typing trajectories at this granularity (released anonymized), and evidence that intent-aware modelling can make XR spelling more forgiving to motor diversity.

Nonspeaking autistic individuals (“nonspeakers”) represent about one-third of the autistic population, yet most lack access to an effective alternative to speech. This lack of effective communication significantly limits their access to educational, social, and employment opportunities. Some nonspeakers have learned to spell words and sentences by pointing to letters on a physical letterboard held in their field of view by a trained human assistant. While effective, this method relies on the assistant for positioning the letterboard, limiting user autonomy and privacy. We report here a system we developed that uses Behavioral Cloning (BC) to automatically and adaptively position a virtual letterboard in Augmented Reality (AR). By observing finger, palm, head, and physical letterboard poses during real-life interactions between a nonspeaker and their assistant, we train a BC Machine Learning (ML) model that can adapt the placement of a virtual letterboard for that user. Results from 11 experiments (3 emulated scenarios and 8 nonspeaking autistic participants) show that our approach can accurately replicate the actions of the human assistant of any given user, outperforming a non-ML baseline personalized placement policy in both positional and rotational accuracies. Further, our novel BC formulation overcomes traditional data-efficiency limitations, allowing us to achieve high accuracy with a modest training effort. This work represents a foundational step toward enabling more autonomous and private communication for nonspeakers.

This paper investigates the feasibility of using video-passthrough Extended Reality (XR) devices to support communication in nonspeaking autistic individuals.Prior XR research with this population has relied on expensive augmented reality headsets and limited interactions to near-hand tapping.We present LetterBox, a novel application developed for video-passthrough XR headsets such as the Meta Quest series, which enables spelling through a "grabsnap-release" interaction.The app supports three immersion levels and includes a dynamic pass-through window tracking caregiver presence.We conducted a study with 19 participants across four

Nonspeaking autistic individuals ("nonspeakers") represent about one-third of the autistic population, and most are never provided with an effective alternative to speech, hindering their educational, employment, and social opportunities. Some individuals have learned to spell words and sentences by pointing to letters on a physical letterboard held vertically in their field of view by a trained human assistant. While this method is effective, nonspeakers have expressed to us a desire to transition towards a more independent communication method that relies less on a human assistant, which would provide them with more autonomy and privacy. Augmented Reality (AR) based communication systems have the potential to address this objective. For example, an AR-based communication system can lessen the reliance on a human assistant by employing a virtual letterboard that is automatically and adaptively placed in a personalized manner that considers a given user’s unique motor skills and movement patterns. In this paper, we explore the use of Behavioural Cloning (BC) to derive such a personalized placement policy. Specifically, we observe finger, palm, head, and physical letterboard poses during real-life interactions between a nonspeaker and their assistant. These observations are then used to train a BC Machine Learning (ML) model that can adapt the placement of a virtual letterboard for that user within an AR environment. Results show that our approach can accurately replicate the actions of the human assistant of any given user, outperforming a non-ML baseline personalized placement policy in both positional and rotational accuracies. This work represents a foundational step toward enabling more autonomous and private communications for nonspeakers, thereby opening up new opportunities for them.

Nonspeaking autistic individuals often face significant inclusion barriers in various aspects of life, mainly due to a lack of effective communication means. Specialized computer software, particularly delivered via Augmented Reality (AR), offers a promising and accessible way to improve their ability to engage with the world. While research has explored near-hand interactions within AR for this population, gaze-based interactions remain unexamined. Given the fine motor skill requirements and potential for fatigue associated with near-hand interactions, there is a pressing need to investigate the potential of gaze interactions as a more accessible option. This paper presents a study investigating the feasibility of eye gaze interactions within an AR environment for nonspeaking autistic individuals. We utilized the HoloLens 2 to create an eye gaze-based interactive system, enabling users to select targets either by fixating their gaze for a fixed period or by gazing at a target and triggering selection with a physical button (referred to as a ‘clicker’). We developed a system called HoloGaze that allows a caregiver to join an AR session to train an autistic individual in gaze-based interactions as appropriate. Using HoloGaze, we conducted a study involving 14 nonspeaking autistic participants. The study had several phases, including tolerance testing, calibration, gaze training, and interacting with a complex interface: a virtual letterboard. All but one participant were able to wear the device and complete the system’s default eye calibration; 10 participants completed all training phases that required them to select targets using gaze only or gaze-click. Interestingly, the 7 users who chose to continue to the testing phase with gaze-click were much more successful than those who chose to continue with gaze alone. We also report on challenges and improvements needed for future gaze-based interactive AR systems for this population. Our findings pave the way for new opportunities for specialized AR solutions tailored to the needs of this under-served and under-researched population.

About one-third of autistic people are nonspeaking, and most are never provided access to an effective alternative to speech. Thoughtfully designed AR applications could provide members of this population with structured learning opportunities, including training on skills that underlie alternative forms of communication. A fundamental step toward creating such opportunities, however, is to investigate nonspeaking autistic people’s ability to tolerate a head-mounted AR device and to interact with virtual objects. We present the first study to examine the usability of an interactive AR-based application by this population. We recruited 17 nonspeaking autistic subjects to play a HoloLens 2 game we developed that involved holographic animations and buttons. Almost all subjects tolerated the device long enough to begin the game, and most completed increasingly challenging tasks that involved pressing holographic buttons. Based on the results, we discuss best practice design and process recommendations. Our findings contradict prevailing assumptions about nonspeaking autistic people and thus open up exciting possibilities for AR-based solutions for this understudied and underserved population.