We present an augmented virtuality system using low-cost or outdated devices as sensors, attached to tangible objects. Five interactions were implemented, featuring rotation, pressing, pulling, pushing and insertion movements, including passive mechanical resistance. Interactions were tied together by a puzzling narrative. Concerning storytelling, in contrast to AR, pure virtual reality offers unlimited visual possibilities and higher flexibility. Augmenting VR with physical objects allows ignoring most details of the physical environment and exploring ‘magical’ interactions. Precise registration was accomplished using wireless tracking system. The main challenge was the absent physical counterparts of the scenario, which was tackled by limiting user's reach to available interactions. Despite Leap Motion's erratic hand tracking, mouse and tablet sensors were precise enough and the augmented environment allowed a high sense of presence.

In this work, we present a hand-based natural interaction that allows performing fundamental actions such as moving or controlling objects and climbing ladders. The setup was restricted to available consumer VR technology, aiming to advance towards a practical unified framework for 3D interaction. The strategy was syncing the closest natural movement allowed by the device with primary task actions, either directly or indirectly, creating hypernatural UIs. The prototype allowed successful completion of the three challenges proposed by the 2018 3DUI Contest, as validated by a preliminary user study with participants from the target audience and also from the general public.