This paper presents an approach for extending 360 VR to support viewpoint translation and collaboration, without increasing the acquisition cost. The approach models the ground under the 360° image acquisition viewpoint with a planar patch, on which the user stands and walks. The planar patch is connected to the background with visual continuity. The approach anchors the user's avatar to the ground patch, and supports collaboration, with multiple avatars standing on the patch. The approach only requires defining two trivial parameters: the approximate acquisition height, and the size of the patch. Therefore, the approach improves the immersive exploration of a real-world scene captured with a 360° image, without increasing the cost of the acquisition of the virtual environment. The approach was evaluated in a controlled user study (N = 30) with five user t asks. The results show that the approach has an advantage over conventional 360 VR in terms of anchoring the user to the ground, of user viewpoint translation, of collaboration, and of accurate integration of stationary and dynamic virtual objects into the 3D scene.

Advances in virtual reality (VR) have reduced experience differentials for users. However, gaps between reality and virtuality persist in tasks that require coupling users’ multimodal physical skills with virtual environments in delicate ways. User embodiment in VR easily breaks when physicality feels inauthentic, especially when users invoke their innate predilection to touch and manipulate things that they encounter. In this research, we examine the potential of forceaware VR interfaces for enabling natural connections to user physicality and evaluate them in high-finesse cases of touch. Combining surface electromyography (SEMG) with visual tracking, we develop an end-to-end learning-based system, ForceSense, to decode users’ dexterous finger forces from their forearm sEMG signals for direct usage in standard VR pipelines. This approach eliminates the need for hand-held tactile equipment, thereby promoting natural embodiment. A series of user studies on manipulation tasks in VR validate that ForceSense is more accurate, robust, and intuitive than alternative solutions. Two proofs-of-concept VR applications, calligraphy and piano playing, demonstrate that the good synergy between visual, auditory, and tactile modalities, as ForceSense affords, has the potential of enhancing users’ task learning performance in VR. Our source code and trained models are released at https://github.com/NYU-ICL/vr-force-aware-multimodal-interface.