In virtual reality (VR), interactions may fail when users encounter new, unknown, or unexpected objects. We propose using feedforward in VR to help users interact with objects by revealing how such objects work. Feedforward lets users know what to do and how to do it by showing the available actions and outcomes before an interaction. In this article, we first chart the design space of feedforward in VR and illustrate how to design feedforward for specific VR interactions. We discuss starting the feedforward, previewing actions and outcomes, and returning the virtual world to its state before the feedforward. Second, we implement three real-world VR applications to show how feedforward can be applied to multistep interactions, perceived interactivity, and discoverability. Third, we conduct an evaluation of the design space with 14 VR experts to understand its usefulness. Finally, we summarize the findings of our work on VR feedforward in 15 guidelines.

A compelling property of virtual reality is that it allows users to interact with objects as they would in the real world. However, such interactions are limited to space within reach. We present Poros, a system that allows users to rearrange space. After marking a portion of space, the distant marked space is mirrored in a nearby proxy. Thereby, users can arrange what is within their reachable space, making it easy to interact with multiple distant spaces as well as nearby objects. Proxies themselves become part of the scene and can be moved, rotated, scaled, or anchored to other objects. Furthermore, they can be used in a set of higher-level interactions such as alignment and action duplication. We show how Poros enables a variety of tasks and applications and also validate its effectiveness through an expert evaluation.

Using microgestures, prior work has successfully enabled gestural interactions while holding objects. Yet, these existing methods are prone to false activations caused by natural finger movements while holding or manipulating the object. We address this issue with SoloFinger, a novel concept that allows design of microgestures that are robust against movements that naturally occur during primary activities. Using a data-driven approach, we establish that single-finger movements are rare in everyday hand-object actions and infer a single-finger input technique resilient to false activation. We demonstrate this concept’s robustness using a white-box classifier on a pre-existing dataset comprising 36 everyday hand-object actions. Our findings validate that simple SoloFinger gestures can relieve the need for complex finger configurations or delimiting gestures and that SoloFinger is applicable to diverse hand-object actions. Finally, we demonstrate SoloFinger’s high performance on commodity hardware using random forest classifiers.

Virtual Reality allows users to embody avatars that do not match their real bodies. Earlier work has selected changes to the avatar arbitrarily and it therefore remains unclear how to change avatars to improve users' performance. We propose a systematic approach for iteratively adapting the avatar to perform better for a given task based on users' performance. The approach is evaluated in a target selection task, where the forearms of the avatar are scaled to improve performance. A comparison between the optimised and real arm lengths shows a significant reduction in average tapping time by 18.7%, for forearms multiplied in length by 5.6. Additionally, with the adapted avatar, participants moved their real body and arms significantly less, and subjective measures show reduced physical demand and frustration. In a second study, we modify finger lengths for a linear tapping task to achieve a better performing avatar, which demonstrates the generalisability of the approach.

Around-device interaction methods expand the available interaction space for mobile devices; however, there is currently no way to simultaneously track a user's input and provide haptic feedback at the tracked point away from the device. We present Magnetips, a simple, mobile solution for around-device tracking and mid-air haptic feedback. Magnetips combines magnetic tracking and electromagnetic feedback that works regardless of visual occlusion, through most common materials, and at a size that allows for integration with mobile devices. We demonstrate: (1) high-frequency around-device tracking and haptic feedback; (2) the accuracy and range of our tracking solution which corrects for the effects of geomagnetism, necessary for enabling mobile use; and (3) guidelines for maximising strength of haptic feedback, given a desired tracking frequency. We present technical and usability evaluations of our prototype, and demonstrate four example applications of its use.

While pressing can enable a wide variety of interesting applications, most press sensing techniques operate only at close distances and rely on fragile electronics. We present EchoTube, a robust, modular, simple, and inexpensive system for sensing low-resolution press events at a distance. EchoTube works by emitting ultrasonic pulses inside a flexible tube which acts as a waveguide and detecting reflections caused by deformations in the tube. EchoTube is deployable in a wide variety of situations: the flexibility of the tubes allows them to be wrapped around and affixed to irregular objects. Because the electronic elements are located at one end of the tube, EchoTube is robust, able to withstand crushing, impacts, water, and other adverse conditions. In this paper, we detail the design, implementation, and theory behind EchoTube; characterize its performance under different configurations; and present a variety of exemplar applications that illustrate its potential.

Recent improvements in ultrasound imaging enable new opportunities for hand pose detection using wearable devices. Ultrasound imaging has remained under-explored in the HCI community despite being non-invasive, harmless and capable of imaging internal body parts, with applications including smart-watch interaction, prosthesis control and instrument tuition. In this paper, we compare the performance of different forearm mounting positions for a wearable ultrasonographic device. Location plays a fundamental role in ergonomics and performance since the anatomical features differ among positions. We also investigate the performance decrease due to cross-session position shifts and develop a technique to compensate for this misalignment. Our gesture recognition algorithm combines image processing and neural networks to classify the flexion and extension of 10 discrete hand gestures with an accuracy above 98%. Furthermore, this approach can continuously track individual digit flexion with less than 5% NRMSE, and also differentiate between digit flexion at different joints.

Wearable devices for activity tracking and gesture recognition have expanded rapidly in recent years. One technique that has shown great potential for this is ultrasonic imaging [10][4]. This technique has been shown to have advantages over other techniques in accuracy, surface area, placement and importantly, continuous finger angle estimations. However, ultrasonic imaging suffers from a couple of issues: First and foremost, the propagation of ultrasound into flesh suffers greatly without a suitable coupling medium; Secondly, the complexity of the driving circuitry for medical grade imaging currently renders a wearable version of this infeasible. This paper aims to address these two problems by finding a rigid coupling medium that lasts for significantly longer periods of time; and devising a new sensor configuration to reduce the device complexity, while still retaining the benefits of the technique. Furthermore, a comparison between high and low frequency systems reveal that different devices can be created with this technique for better resolution or convenience respectively.

Gestures have become an important tool for natural interaction with computers and thus several wearables have been developed to detect hand gestures. However, many existing solutions are unsuitable for practical use due to low accuracy, high cost or poor ergonomics. We present SensIR, a bracelet that uses near-infrared sensing to infer hand gestures. The bracelet is composed of pairs of infrared emitters and receivers that are used to measure both the transmission and reflection of light through/off the wrist. SensIR improves the accuracy of existing infrared gesture sensing systems through the key idea of taking measurements with all possible combinations of emitters and receivers. Our study shows that SensIR is capable of detecting 12 discrete gestures with 93.3% accuracy. SensIR has several advantages compared to other systems such as high accuracy, low cost, robustness against bad skin coupling and thin form-factor.

Practical wearable gesture tracking requires that sensors align with existing ergonomic device forms. We show that combining EMG and pressure data sensed only at the wrist can support accurate classification of hand gestures. A pilot study with unintended EMG electrode pressure variability led to exploration of the approach in greater depth. The EMPress technique senses both finger movements and rotations around the wrist and forearm, covering a wide range of gestures, with an overall 10-fold cross validation classification accuracy of 96%. We show that EMG is especially suited to sensing finger movements, that pressure is suited to sensing wrist and forearm rotations, and their combination is significantly more accurate for a range of gestures than either technique alone. The technique is well suited to existing wearable device forms such as smart watches that are already mounted on the wrist.