Image-based scene understanding allows Augmented Reality (AR) systems to provide contextual visual guidance in unprepared, real-world environments. While effective on video see-through (VST) head-mounted displays (HMDs), such methods suffer on optical see-through (OST) HMDs due to misregistration between the world-facing camera and the user's eye perspective. To approximate the user's true eye view, we implement and evaluate three software-based eye-perspective rendering (EPR) techniques on a commercially available, untethered OST HMD (Microsoft HoloLens 2): (1) Plane-Proxy EPR, projecting onto a fixed-distance plane; (2) Mesh-Proxy EPR, using SLAM-based reconstruction for projection; and (3) Gaze-Proxy EPR, a novel eye-tracking-based method that aligns the projection with the user's gaze depth. A user study on real-world tasks underscores the importance of accurate EPR and demonstrates gaze-proxy as a lightweight alternative to geometry-based methods. We release our EPR framework as open source.

Optical see-through (OST) head-mounted displays (HMDs) enable users to experience Augmented Reality (AR) support in the form of helpful real-world annotations. Unfortunately, the blend of the environment with virtual augmentations due to semitransparent OST displays often deteriorates the contrast and legibility of annotations. View management algorithms adapt the annotations’ layout to improve legibility based on real-world information, typically captured by built-in HMD cameras. However, the camera views are different from the user’s view through the OST display which decreases the final layout quality. We present eye-perspective view management that synthesizes high-fidelity renderings of the user’s view to optimize annotation placement. Our method significantly improves over traditional camera-based view management in terms of annotation placement and legibility. Eye-perspective optimizations open up opportunities for further research on use cases relying on the user’s true view through OST HMDs.

Increased levels of interactivity and multi-sensory stimulation have been shown to enhance the immersion of Virtual Reality experiences. We present the AirRes mask that enables users to utilize their breathing for precise natural interactions with the virtual environment without suffering from limitations of the sensing equipment such as motion artifacts. Furthermore, the AirRes mask provides breathing resistance as novel output modality that can be adjusted in real-time by the application. In a user study, we demonstrate the mask’s precision measurements for interaction as well as its ability to use breathing resistance to communicate contextual information such as adverse environmental conditions that affect the user’s virtual avatar. Our results show that the AirRes mask enhances virtual experiences and has the potential to create more immersive scenarios for applications by enforcing the perception of danger or improving situational awareness in training simulations, or for psychotherapy by providing additional physical stimuli.

Patients with Frozen Shoulder suffer from a decreased mobility and pain. Exercise-based physiotherapy is a common treatment and patients mostly perform the exercises at home. Correct exercise performance and compliance are the main issues in home-based therapy of Frozen Shoulder patients. To support patients diagnosed with Frozen Shoulder, a multimodal 3D smartphone app was designed, developed and evaluated. Additional to ten potential users, one physician, five physiotherapists, three computer scientists, two 3D artists and one HCI specialist were involved in the co-creation process. The app was evaluated by five patients during a three-week pilot study, which showed the feasibility of our approach. Exercise correctness, usage of multimodal instructions and user satisfaction were analyzed. Exercise correctness was nearly perfect and the interactive 3D animation was used for exercise instructions. Satisfaction using the app was rated very high according to SUS score. The results confirm that the co-creation process led to an effective, highly satisfactory and actually used system.