Smart glasses enhance interactions with the environment by using head-mounted cameras to observe the user’s viewpoint, but lack the visual feedback used for common interactions. We introduce “Gazeify then Voiceify”, a multimodal approach allowing object selection via gaze and voice using displayless smart glasses. Users can select a physical object with their gaze, and the system generates a digital mask and a voice description of the object’s semantics. Users can further correct errors through free-form conversation. To demonstrate our approach, we develop an interactive system by integrating advanced object segmentation and detection with a visual-language model. User studies reveal that participants achieve correct gaze selection in 53% of the task trials and use voice disambiguation to correct 58% remaining errors. Participants also rated the system as likable, useful and easy to use.

We explored how interaction performance is affected by multi-depth VR targeting and button selection using two targeting methods: eye tracking with no UX modifications and feedback, or the controller with a visible cursor for targeting. Selections happened on a controller button press for both targeting modalities. Targets had a diameter of either 3, 4, or 5 degrees, placed in depths between 0.3 m-5m. When comparing conditions of a 1 m single depth vs. multi-depth environment, the eyes were less affected by depth than the controller. We found that performance decreased in multi-depth scenarios on targeting and selection for the controller as measured by Throughput (22% decrease), Movement Time (31% increase), and Misses (66% increase). Depth also affected eye tracking significantly, but to a lesser degree, for Throughput (4% decrease) and Movement Time (6% increase) but not Misses (5% increase). The eyes outperformed the controller in multi-depth scenarios, as measured by Throughput (2.86 bits/s vs. 2.56 bits/s), and were similar in Movement Time (1.10s vs. 1.10s) but had the most Misses (21% vs. 9%). Our study also shows that selecting consecutive targets that come closer to the user is more difficult than those that diverge away from the user, and that targets with larger depth distances take longer to select. Overall, this study provides further supporting evidence that eye tracking can play an important role in 3D interactions.

Can eye tracking enable VR users to target and select elements at par or better than controller or head-based targeting? We explored visual feedback methods (none, cursor, outline and resize) for gaze targeting with a button press for selection, and an ecologically valid representation of commercially available user interfaces with a body-locked, grid-based design. Forty participants interacted with a 5x5 square element grid, and elements subtended either 3-, 6- or 9-degrees of visual angle. If the participant looked out of the grid boundary, on button press, we chose to select the last targeted element, but no other algorithms to enhance performance were employed. We also assessed signal quality requirements with a fixed offset 1.5-degree accuracy degradation. Participants completed 36 blocks and in each, targeted and selected 15 successive elements. We found that gaze targeting, with appropriate feedback methods and a button press, can perform at par or better than the controller in cases intended to replicate targeting and selecting in world- or body-locked paradigms in AR/VR. We anticipate that with design improvements or algorithmic mitigations that this can improve significantly.

Gaze-based user interfaces and interactions are becoming more prevalent in augmented and virtual reality (AR/VR). The effectiveness of eye tracking for interaction depends on its quality. Many studies discuss eye tracking as an input and interaction modality but do not provide details about eye tracking quality, making it difficult to compare findings. Here we implement a framework to degrade accuracy error with the user in the loop. We then approximate calibration error, with those degradations applied in each block to provide an “Effective Gaze Error.” Participants selected single targets (3° or 5° diameter) using an eye tracking sampling frequency and display rate of 120 Hz. Higher “Effective Gaze Error” on smaller targets resulted in decreased human performance and subjective evaluations. Our experiment framework and results provide a starting point for future studies assessing how gaze accuracy degradation impacts performance, beyond interactions tasks.

In this study, we establish a much-needed baseline for evaluating eye tracking interactions using an eye tracking enabled Meta Quest 2 VR headset with 30 participants. Each participant went through 1098 targets using multiple conditions representative of AR/VR targeting and selecting tasks, including both traditional standards and those more aligned with AR/VR interactions today. We use circular white world-locked targets, and an eye tracking system with sub-1-degree mean accuracy errors running at approximately 90Hz. In a targeting and button press selection task, we, by design, compare completely unadjusted, cursor-less, eye tracking with controller and head tracking, which both had cursors. Across all inputs, we presented targets in a configuration similar to the ISO 9241-9 reciprocal selection task and another format with targets more evenly distributed near the center. Targets were laid out either flat on a plane or tangent to a sphere and rotated toward the user. Even though we intended this to be a baseline study, we see unmodified eye tracking, without any form of a cursor, or feedback, outperformed the head by 27.9% and performed comparably to the controller (5.63% decrease) in throughput. Eye tracking had improved subjective ratings relative to head in Ease of Use, Adoption, and Fatigue (66.4%, 89.8%, and 116.1 % improvements, respectively) and had similar ratings relative to the controller (reduction by 4.2%, 8.9%, and 5.2% respectively). Eye tracking had a higher miss percentage than controller and head (17.3% vs 4.7% vs 7.2% respectively). Collectively, the results of this baseline study serve as a strong indicator that eye tracking, with even minor sensible interaction design modifications, has tremendous potential in reshaping interactions in next-generation AR/VR head mounted displays.