Nonspeaking autistic individuals often build expressive communication through spelling on physical letterboards, but skilled one-on-one support is difficult to scale. XR letterboards could expand access, yet commodity near-hand press recognizers are brittle under the motor strategies common in nonspeaking typers (e.g., brief pokes, glides, folded-finger taps), leading to missed interactions that disrupt fluency and trust. We present SAGE (Spatial Anticipation of Gestural Events), a lightweight intent-inference layer that recovers intended selections even when no device press is registered. SAGE first fits a per-user, per-key spatial prior from historical device-registered taps, then fuses this prior at runtime with fingertip proximity and wrist-anchored pointing, using temporal peak detection to infer both press timing and letter identity. We evaluate SAGE on a HoloLens 2 AR letterboard dataset from four nonspeaking participants (244,620 hand-tracking samples) and compare against the headset recognizer and two physics-based replay baselines. On known taps, SAGE improves attribution quality (F1=0.91 vs. best baseline 0.52); on missed attempts, it recovers 71% of unregistered taps with correct letter identity for all recovered events. We contribute SAGE, the first dataset of nonspeaking XR typing trajectories at this granularity (released anonymized), and evidence that intent-aware modelling can make XR spelling more forgiving to motor diversity.

Nonspeaking autistic individuals (“nonspeakers”) represent about one-third of the autistic population, yet most lack access to an effective alternative to speech. This lack of effective communication significantly limits their access to educational, social, and employment opportunities. Some nonspeakers have learned to spell words and sentences by pointing to letters on a physical letterboard held in their field of view by a trained human assistant. While effective, this method relies on the assistant for positioning the letterboard, limiting user autonomy and privacy. We report here a system we developed that uses Behavioral Cloning (BC) to automatically and adaptively position a virtual letterboard in Augmented Reality (AR). By observing finger, palm, head, and physical letterboard poses during real-life interactions between a nonspeaker and their assistant, we train a BC Machine Learning (ML) model that can adapt the placement of a virtual letterboard for that user. Results from 11 experiments (3 emulated scenarios and 8 nonspeaking autistic participants) show that our approach can accurately replicate the actions of the human assistant of any given user, outperforming a non-ML baseline personalized placement policy in both positional and rotational accuracies. Further, our novel BC formulation overcomes traditional data-efficiency limitations, allowing us to achieve high accuracy with a modest training effort. This work represents a foundational step toward enabling more autonomous and private communication for nonspeakers.

This paper investigates the feasibility of using video-passthrough Extended Reality (XR) devices to support communication in nonspeaking autistic individuals.Prior XR research with this population has relied on expensive augmented reality headsets and limited interactions to near-hand tapping.We present LetterBox, a novel application developed for video-passthrough XR headsets such as the Meta Quest series, which enables spelling through a "grabsnap-release" interaction.The app supports three immersion levels and includes a dynamic pass-through window tracking caregiver presence.We conducted a study with 19 participants across four

Many nonspeaking autistic individuals rely on Communication and Regulation Partners (CRPs) to develop spelling-based communication using physical letterboards, but this support is often geographically inaccessible.We developed a remote presence system using Augmented Reality (AR) to enable immersive, collaborative spelling instruction.The system features holographic letterboards and fully embodied avatars with real-time head and hand tracking, allowing remote interaction between students and CRPs.In a study with 18 nonspeaking autistic participants, 15 (83%) successfully completed avatar-supported sessions.Interaction was higher, and participants reported a preference for the avatar condition over voice-only support.These findings demonstrate the feasibility of avatar-based AR telepresence for remote communication training.The system provides a demonstration of AR-supported interaction designed with nonspeaking autistic individuals-an underrepresented group in HCI-and offers design insights for inclusive telepresence technologies that address geographic and accessibility barriers.

About one-third of autistic individuals are nonspeaking, i.e., they cannot use speech to convey their thoughts reliably. Many in this population communicate via spelling, a process in which they point to letters on a letterboard held upright in their field of view by a trained Communication and Regulation Partner (CRP). This paper focuses on transitioning such individuals to more independent, digital spelling that requires less support from the CRP, a goal most nonspeakers we consulted with desire. To enable this transition, we followed an approach that mimics an environment familiar to the nonspeaker and that harnesses the skills they already possess from physical letterboard training. Using this approach, we developed HoloBoard, a system that allows a nonspeaker, their CRP, and others, e.g., researchers, to share a common Augmented Reality (AR) environment containing a virtual letterboard. We configured the system to offer a brief (less than 10 minutes, on average) training module with graduated spelling tasks on the virtual letterboard. In a study involving 23 participants, 16 completed the entire module. These participants were able to spell words on the virtual letterboard without the CRP holding that board, an outcome we had not expected. When offered the opportunity to continue interacting with the virtual letterboard after the training module, 14 performed more complicated tasks than we had anticipated, spelling full sentences, or even offering feedback on the HoloBoard using solely the virtual board. Furthermore, five of these participants used the system solo, i.e., with the CRP and researchers absent from the virtual environment. These results suggest that training with the HoloBoard can lay the foundation for more independent communication, providing new social and educational opportunities for this marginalized population.

Nonspeaking autistic individuals ("nonspeakers") represent about one-third of the autistic population, and most are never provided with an effective alternative to speech, hindering their educational, employment, and social opportunities. Some individuals have learned to spell words and sentences by pointing to letters on a physical letterboard held vertically in their field of view by a trained human assistant. While this method is effective, nonspeakers have expressed to us a desire to transition towards a more independent communication method that relies less on a human assistant, which would provide them with more autonomy and privacy. Augmented Reality (AR) based communication systems have the potential to address this objective. For example, an AR-based communication system can lessen the reliance on a human assistant by employing a virtual letterboard that is automatically and adaptively placed in a personalized manner that considers a given user’s unique motor skills and movement patterns. In this paper, we explore the use of Behavioural Cloning (BC) to derive such a personalized placement policy. Specifically, we observe finger, palm, head, and physical letterboard poses during real-life interactions between a nonspeaker and their assistant. These observations are then used to train a BC Machine Learning (ML) model that can adapt the placement of a virtual letterboard for that user within an AR environment. Results show that our approach can accurately replicate the actions of the human assistant of any given user, outperforming a non-ML baseline personalized placement policy in both positional and rotational accuracies. This work represents a foundational step toward enabling more autonomous and private communications for nonspeakers, thereby opening up new opportunities for them.

Nonspeaking autistic individuals often face significant inclusion barriers in various aspects of life, mainly due to a lack of effective communication means. Specialized computer software, particularly delivered via Augmented Reality (AR), offers a promising and accessible way to improve their ability to engage with the world. While research has explored near-hand interactions within AR for this population, gaze-based interactions remain unexamined. Given the fine motor skill requirements and potential for fatigue associated with near-hand interactions, there is a pressing need to investigate the potential of gaze interactions as a more accessible option. This paper presents a study investigating the feasibility of eye gaze interactions within an AR environment for nonspeaking autistic individuals. We utilized the HoloLens 2 to create an eye gaze-based interactive system, enabling users to select targets either by fixating their gaze for a fixed period or by gazing at a target and triggering selection with a physical button (referred to as a ‘clicker’). We developed a system called HoloGaze that allows a caregiver to join an AR session to train an autistic individual in gaze-based interactions as appropriate. Using HoloGaze, we conducted a study involving 14 nonspeaking autistic participants. The study had several phases, including tolerance testing, calibration, gaze training, and interacting with a complex interface: a virtual letterboard. All but one participant were able to wear the device and complete the system’s default eye calibration; 10 participants completed all training phases that required them to select targets using gaze only or gaze-click. Interestingly, the 7 users who chose to continue to the testing phase with gaze-click were much more successful than those who chose to continue with gaze alone. We also report on challenges and improvements needed for future gaze-based interactive AR systems for this population. Our findings pave the way for new opportunities for specialized AR solutions tailored to the needs of this under-served and under-researched population.

About one-third of autistic people are nonspeaking, and most are never provided access to an effective alternative to speech. Thoughtfully designed AR applications could provide members of this population with structured learning opportunities, including training on skills that underlie alternative forms of communication. A fundamental step toward creating such opportunities, however, is to investigate nonspeaking autistic people’s ability to tolerate a head-mounted AR device and to interact with virtual objects. We present the first study to examine the usability of an interactive AR-based application by this population. We recruited 17 nonspeaking autistic subjects to play a HoloLens 2 game we developed that involved holographic animations and buttons. Almost all subjects tolerated the device long enough to begin the game, and most completed increasingly challenging tasks that involved pressing holographic buttons. Based on the results, we discuss best practice design and process recommendations. Our findings contradict prevailing assumptions about nonspeaking autistic people and thus open up exciting possibilities for AR-based solutions for this understudied and underserved population.

Approximately one-third of individuals with autism are nonspeaking: They cannot communicate effectively using speech. Some traditional accounts suggest that these individuals cannot talk because they lack the symbolic capacity for language. And yet, recent studies have shown that these individuals’ cognitive abilities are vastly underestimated by standardized tests, and that difficulties with motor skills and movement contribute to their difficulty with speech. One consequence of the traditional accounts of nonspeaking autism is that life skills (rather than academic content) tend to be emphasized in schooling. Without access to meaningful academic content, their educational and vocational opportunities are significantly limited. Recent studies have proposed the use of head-mounted Augmented Reality (AR) applications as a means of providing engaging, customizable, and age-appropriate content to this population. Specifically, such applications can address the unique sensory and motor needs of nonspeaking autistic students, e.g., allow them to move freely around the room as they interact with lessons in the application. This paper describes the design and evaluation of the first AR application aimed to facilitate tailored educational experiences for nonspeaking autistic students. After extensive consultations with nonspeaking people, parents, and professionals, we developed our application to run on HoloLens 2 offering lessons and multiple-choice comprehension and spelling questions. We conducted a study involving five nonspeaking autistic participants and two specialized educators. Through a design critique process and an iterative design refinement approach, we show that most of our participants successfully interacted with the application and completed different types of lesson tasks. Based on quantitative data from the study sessions and qualitative feedback from participants and educators, we provide recommendations for UI and UX design that will promote the development and use of such software for this under-served and under-researched population.