We explored how interaction performance is affected by multi-depth VR targeting and button selection using two targeting methods: eye tracking with no UX modifications and feedback, or the controller with a visible cursor for targeting. Selections happened on a controller button press for both targeting modalities. Targets had a diameter of either 3, 4, or 5 degrees, placed in depths between 0.3 m-5m. When comparing conditions of a 1 m single depth vs. multi-depth environment, the eyes were less affected by depth than the controller. We found that performance decreased in multi-depth scenarios on targeting and selection for the controller as measured by Throughput (22% decrease), Movement Time (31% increase), and Misses (66% increase). Depth also affected eye tracking significantly, but to a lesser degree, for Throughput (4% decrease) and Movement Time (6% increase) but not Misses (5% increase). The eyes outperformed the controller in multi-depth scenarios, as measured by Throughput (2.86 bits/s vs. 2.56 bits/s), and were similar in Movement Time (1.10s vs. 1.10s) but had the most Misses (21% vs. 9%). Our study also shows that selecting consecutive targets that come closer to the user is more difficult than those that diverge away from the user, and that targets with larger depth distances take longer to select. Overall, this study provides further supporting evidence that eye tracking can play an important role in 3D interactions.

Can eye tracking enable VR users to target and select elements at par or better than controller or head-based targeting? We explored visual feedback methods (none, cursor, outline and resize) for gaze targeting with a button press for selection, and an ecologically valid representation of commercially available user interfaces with a body-locked, grid-based design. Forty participants interacted with a 5x5 square element grid, and elements subtended either 3-, 6- or 9-degrees of visual angle. If the participant looked out of the grid boundary, on button press, we chose to select the last targeted element, but no other algorithms to enhance performance were employed. We also assessed signal quality requirements with a fixed offset 1.5-degree accuracy degradation. Participants completed 36 blocks and in each, targeted and selected 15 successive elements. We found that gaze targeting, with appropriate feedback methods and a button press, can perform at par or better than the controller in cases intended to replicate targeting and selecting in world- or body-locked paradigms in AR/VR. We anticipate that with design improvements or algorithmic mitigations that this can improve significantly.

All-day augmented reality (AR) requires contextually-aware artificial intelligence (AI) models that excel across diverse daily contexts. Eye tracking could be a key source of information about user context and intention. However, such models using gaze sometimes struggle to outperform egocentric video-based baseline models. We propose that learning representations of scanpath history in a perceptually-relevant state space may solve this problem. However, scanpaths are often assumed to obey a Markovian assumption, i.e., only the current and previous fixation matter. In a user study (30 participants; 26.2 hours total), we analyzed scanpaths during nine everyday tasks and identified long-term temporal dependencies, with an average timescale of four fixations (2 seconds) into the past (i.e., violating the Markovian assumption). We discovered substantial task-specific variations in these dependencies. This confirms that scanpaths contain stereotyped “motifs” with context-dependent lengths/timescales. We discuss the implications for designing contextual AI models for AR applications.

Gaze-based user interfaces and interactions are becoming more prevalent in augmented and virtual reality (AR/VR). The effectiveness of eye tracking for interaction depends on its quality. Many studies discuss eye tracking as an input and interaction modality but do not provide details about eye tracking quality, making it difficult to compare findings. Here we implement a framework to degrade accuracy error with the user in the loop. We then approximate calibration error, with those degradations applied in each block to provide an “Effective Gaze Error.” Participants selected single targets (3° or 5° diameter) using an eye tracking sampling frequency and display rate of 120 Hz. Higher “Effective Gaze Error” on smaller targets resulted in decreased human performance and subjective evaluations. Our experiment framework and results provide a starting point for future studies assessing how gaze accuracy degradation impacts performance, beyond interactions tasks.

A key feature of augmented reality (AR) is the ability to display virtual content that appears stationary as users move throughout the physical world (‘world-locked rendering’). Imperfect world-locked rendering gives rise to perceptual artifacts that can negatively impact user experience. One example is random variation in the position of virtual objects that are intended to be stationary (jitter’). The human visual system is highly attuned to detect moving objects, and moreover it can disambiguate between the retinal velocities that arise from object motion and self-motion, respectively. In this study, we investigated how the perceptibility of AR object jitter varies as a function of user self-motion. Using a commercially available AR HMD to display a 3D textured cube, we measured sensitivity to added jitter versus a no-jitter reference using a two-interval forced choice task. Three user motion conditions (stationary, head rotation, and walking) and three object placement conditions (floating in free space, on a desk, and against a wall) were tested in a full factorial design. We hypothesized that (1) as users move their head and eyes during self-motion, their sensitivity to jitter will decrease, due to added retinal velocity; and (2) rendering virtual objects near physical surfaces will increase sensitivity to jitter, by providing proximal veridical visual cues. Psychometric thresholds indicated that users were significantly less sensitive to jitter during self-motion than when they were stationary, consistent with hypothesis (1). Users were also more sensitive to jitter in one of the two object placement conditions, providing partial support for hypothesis (2). To generalize beyond distinct user motion and object placement conditions, we also analyzed eye tracking data. The amount of retinal slip (i.e. how much gaze drifted across the virtual object) predicted jitter thresholds better than recorded head movements alone, suggesting a retinally-driven decrease in jitter sensitivity during self-motion. These results can be used to inform requirements for AR world-locked rendering systems, as well as how these may be updated dynamically using online measurement of user head and eye movements.

In this study, we establish a much-needed baseline for evaluating eye tracking interactions using an eye tracking enabled Meta Quest 2 VR headset with 30 participants. Each participant went through 1098 targets using multiple conditions representative of AR/VR targeting and selecting tasks, including both traditional standards and those more aligned with AR/VR interactions today. We use circular white world-locked targets, and an eye tracking system with sub-1-degree mean accuracy errors running at approximately 90Hz. In a targeting and button press selection task, we, by design, compare completely unadjusted, cursor-less, eye tracking with controller and head tracking, which both had cursors. Across all inputs, we presented targets in a configuration similar to the ISO 9241-9 reciprocal selection task and another format with targets more evenly distributed near the center. Targets were laid out either flat on a plane or tangent to a sphere and rotated toward the user. Even though we intended this to be a baseline study, we see unmodified eye tracking, without any form of a cursor, or feedback, outperformed the head by 27.9% and performed comparably to the controller (5.63% decrease) in throughput. Eye tracking had improved subjective ratings relative to head in Ease of Use, Adoption, and Fatigue (66.4%, 89.8%, and 116.1 % improvements, respectively) and had similar ratings relative to the controller (reduction by 4.2%, 8.9%, and 5.2% respectively). Eye tracking had a higher miss percentage than controller and head (17.3% vs 4.7% vs 7.2% respectively). Collectively, the results of this baseline study serve as a strong indicator that eye tracking, with even minor sensible interaction design modifications, has tremendous potential in reshaping interactions in next-generation AR/VR head mounted displays.

When using a see-through augmented reality head-mounted display system (AR HMD), a user’s perception of virtual content may be degraded by a variety of perceptual artifacts resulting from the architecture of rendering and display pipelines. In particular, virtual content that is rendered to appear stationary in the real world (worldlocked) can be susceptible to spatial and temporal 3D position errors. A subset of these errors, termed jitter, result from mismatches between the spatial localization, rendering, and display pipelines, and can manifest as perceived motion of intended-to-be stationary content. Here, we employ psychophysical methods to quantify the perceptibility of jitter artifacts in an AR HMD. For some viewing conditions, participants perceived jitter that was smaller than the pixel pitch of the testbed (i.e., subpixel jitter). In general, we found that jitter perceptibility increased as viewing distance increased and decreased as background luminance increased. We did not find that the contrast ratio of virtual content, age, or experience with AR/VR modulatedjitter perceptibility. Taken together, this study quantifies the degree of jitter that a user can perceive in an AR HMD and demonstrates that it is critical to consider the capabilities and limits of the human visual system when designing the next generation of spatial computing platforms.

Eye gaze as an input method has been studied since the 1990s, to varied results: some studies found gaze to be more efficient than traditional input methods like a mouse, others far behind. Comparisons are often backed up by Fitts' Law without explicitly acknowledging the ballistic nature of saccadic eye movements. Using a vision science-inspired model, we here show that a Fitts'-like distribution of movement times can arise due to the execution of secondary saccades, especially when targets are small. Study participants selected circular targets using gaze. Seven different target sizes and two saccade distances were used. We then determined performance across target sizes for different sampling windows ("dwell times") and predicted an optimal dwell time range. Best performance was achieved for large targets reachable by a single saccade. Our findings highlight that Fitts' Law, while a suitable approximation in some cases, is an incomplete description of gaze interaction dynamics.