Current Mixed Reality (MR) headsets are typically controlled by mid-air hand inputs, which require significant arm effort and consequently reduce input speed and accuracy during extended sessions. We propose BlanchTouch, a research prototype that explores an alternative touch input approach for MR by appropriating uninstrumented flat surfaces as interactive areas, with reduced arm effort. It incorporates a CNN model that detects the fingertip’s pressure-induced blanching phenomenon when a firm touch is applied to hard surfaces, serving as a confirmation of selection. Additionally, hand tracking via the headset is employed to determine the fingertip’s position, thereby providing targeting information for virtual buttons or objects. BlanchTouch operates using only the headset’s built-in cameras and sensors, without requiring additional hardware or prior surface calibration, enabling walk-up use in unfamiliar environments. We conducted two user studies to evaluate the system and compare its performance against a baseline (a real touchscreen) and the state-of-the-art mid-air interaction. Results demonstrated that BlanchTouch achieved an average input positional error of 5.33mm and a recall rate of 97.88%. Furthermore, the findings indicate that BlanchTouch enables competitive input speeds, alleviates arm strain, and delivers satisfactory feedback for each touch.

Locomotion techniques are essential to enhance the sense of presence or "being there" in virtual reality. This research proposes a novel technique, "Daddy Long Legs" (DLL), for navigating medium-scale virtual environments. DLL builds on principles from Seven-league Boots (7LB) and Ground-level Scaling (GLS), aiming to enhance positional accuracy by offering an elevated perspective while preserving immersion through maintaining the user's interpupillary distance. A user study (n = 24) compared three methods across comfort, embodiment, workload, and walking behavior. Participants completed walking tasks in a virtual garden, followed by questionnaires and interviews to capture their experiences. Results show that DLL outperformed 7LB in all measures. However, GLS was noted for its near-natural walking behavior, received the most favorable feedback, and was the preferred locomotion method.

Immersive knowledge spaces like museums or cultural sites are often explored by traversing pre-defined paths that are curated to unfold a specific educational narrative. To support this type of guided exploration in VR, we present a semi-automated, hands-free path traversal technique based on teleportation that features a slow-paced interaction workflow targeted at fostering knowledge acquisition and maintaining spatial awareness. In an empirical user study with 34 participants, we evaluated two variations of our technique, differing in the presence or absence of intermediate teleportation points between the main points of interest along the route. While visiting additional intermediate points was objectively less efficient, our results indicate significant benefits of this approach regarding the user’s spatial awareness and perception of interface dependability. However, the user’s perception of flow, presence, attractiveness, perspicuity, and stimulation did not differ significantly. The overall positive reception of our approach encourages further research into semi-automated locomotion based on teleportation and provides initial insights into the design space of successful techniques in this domain.

Abstract Passive haptics have long been used to enhance the user’s experience in virtual reality (VR). However, creating props to be used in a virtual environment can be a complicated and lengthy process. Current research looks to create passive haptic props based on the layout of, or objects in, the user’s real environment. However, we identify three key limitations of current research. Firstly, procedural generation introduces many unknown variables into the design process, which complicates applying such techniques to scenarios requiring knowledge of the virtual environment’s layout ahead of time. Furthermore, such techniques limit the size and dimensions of the virtual space to that of the real space. Lastly, current research necessitates pre-scanning or real-time scanning of the user’s real environment, often requiring specialist equipment and expertise, thus limiting its generalisability. This research proposes P rop O riented W orld R otation , a technique that attempts to answer the aforementioned limitations and simplify the process of adding haptic feedback to VR applications. We implemented this technique in a demonstration game and give an overview of the steps taken to apply the technique in a real context. We analysed the demonstration system’s performance and conducted an initial user evaluation in three different physical environments. While our stress test of the system’s performance highlights the necessity for certain optimisations in complex environments, our initial user feedback suggests that users experienced a stronger sense of presence and feelings of safety in our passive haptics-enhanced environment. Hence, we conclude that our proposal has the potential to enhance experiences in VR with haptic feedback.

This paper presents a novel solution for estimating simulator sickness in HMDs using machine learning and 3D motion data, informed by user-labeled simulator sickness data and user analysis. We conducted a novel VR user study, which decomposed motion data and used an instant dial-based sickness scoring mechanism. We were able to emulate typical VR usage and collect user simulator sickness scores. Our user analysis shows that translation and rotation differently impact user simulator sickness in HMDs. In addition, users’ demographic information and self-assessed simulator sickness susceptibility data are collected and show some indication of potential simulator sickness. Guided by the findings from the user study, we developed a novel deep learning-based solution to better estimate simulator sickness with decomposed 3D motion features and user profile information. The model was trained and tested using the 3D motion dataset with user-labeled simulator sickness and profiles collected from the user study. The results show higher estimation accuracy when using the 3D motion data compared with methods based on optical flow extracted from the recorded video, as well as improved accuracy when decomposing the motion data and incorporating user profile information.

Due to the level of immersion and differences in the user interface, there can be a very large discrepancy in the user experience between users in immersive systems and non-immersive systems when playing games together. To investigate the impact of the cross-reality experience, which refers to the asymmetric use of eXtended Reality, we aim to understand the different affordances and experiences in an asymmetric setup, where one participant uses a desktop setup with a mouse and keyboard, and one uses a virtual reality (VR) headset and controller in two different task modes, Competition or Collaboration. In our research, a pair of participants played a game in real-time, using either the VR setup or the desktop setup. In Competition mode, the two participants were asked to defeat each other. In Collaboration mode, the pair of participants played as a team and were asked to defeat a pair of AI enemies. Our results show the VR group reported a better gaming experience and perceptual responses compared to the desktop group regardless of game mode, but that the desktop group showed superior gaming performance compared to the VR group in Competition mode.

Despite the increasing popularity of VR games, one factor hindering the industry’s rapid growth is motion sickness experienced by the users. Symptoms such as fatigue and nausea severely hamper the user experience. Machine Learning methods could be used to automatically detect motion sickness in VR experiences, but generating the extensive labeled dataset needed is a challenging task. It needs either very time consuming manual labeling by human experts or modification of proprietary VR application source codes for label capturing. To overcome these challenges, we developed a novel data collection tool, VRhook, which can collect data from any VR game without needing access to its source code. This is achieved by dynamic hooking, where we can inject custom code into a game’s run-time memory to record each video frame and its associated transformation matrices. Using this, we can automatically extract various useful labels such as rotation, speed, and acceleration. In addition, VRhook can blend a customized screen overlay on top of game contents to collect self-reported comfort scores. In this paper, we describe the technical development of VRhook, demonstrate its utility with an example, and describe directions for future research.

This paper investigates the influence of floor-vibration tactile feedback on immersed users. Under symmetric and asymmetric tactile sensory cue conditions, we explore how multi-user Virtual Reality (VR) experiences are impacted by these cues in terms of illusion and coherence. Based on the reported positive impact of tactile cues in solo VR experiences, we posit that if context-matched perceptual tactile feedback is exchanged between users, they will report a significantly enhanced VR experience compared to not receiving the sensory stimuli, even within the same immersive VR experience. With our custom-built, computer-controlled vibration floor, we implemented a cannonball shooting game for two physically-separated players. In the VR game, the two players shoot cannonballs to destroy their opponent’s protective wall and cannon, while the programmed floor platform generates vertical vibrations depending on the experimental condition. We used a mixed-factorial design with four conditions for each pair of participants: 1) both A and B had vibration, and 2) neither A nor B had vibration (the Symmetric group), or 3) A had vibration, but B did not, and 4) B had vibration, but A did not (the Asymmetric group). We collected subjective and objective data for variables previously shown to be related to levels of illusion, coherence, and usability, including Presence, Co-Presence, Social Presence, Plausibility Illusion, Engagement, Embodiment, Coherence, Gaming Performance, and Overall Preference. A total of 39 pairs of participants were involved in the study. We found statistically significant differences for the vibration conditions on Co-Presence, Social Presence, Engagement, and Coherence, and for the symmetric conditions on the Plausibility Illusion and Coherence, but only with trivial or small effect sizes. The results indicate that vibration provided to a pair of game players in immersive VR can significantly enhance the VR experience, but sensory symmetry does not guarantee improved gaming performance.

As a result of urban concentration, colonisation, increased physical distance from tribal homelands, and globalisation, many indigenous people, including Māori, are seeking digital solutions to connect to their culture and identity. Through co-design, close collaboration with our indigenous partners, and careful consideration of cultural context, we show that mixed reality experiences can be an effective mechanism for the growing diaspora of Māori to access and experience their language, genealogy, families, histories and knowledge. Inductive analysis of semi-structured interviews highlights the importance of cultural values and context in this experience, and confirms that this approach can support connections to community and culture. Our work is deeply embedded in a particular indigenous group’s context and culture, but we believe that it holds important lessons that can generalise to other groups. In particular, collaborative co-design and recognition of cultural values throughout the project are essential for producing experiences that meet the needs of specific communities, and that reflect and respect their culture and worldview.

In this paper, we present a system that allows developers to add passive haptic feedback into their virtual reality applications by making use of existing physical objects in the user’s real environment. Our approach has minimal dependence on procedural generation and does not limit the virtual space to the dimensions of the physical play-area.

Physical face-to-face collaboration with someone gives a higher-quality experience compared to mediated communication options, such as a phone- or video-based chat. Participants can share rich sensory cues to multiple human senses in a physical space. Also, the perceptual sensing of the surrounding environment including other peoples' reactions can influence human communication and emotion, and thus collaborative performance. Shared spaces in virtual environments provide degraded sensory experiences because most commercial virtual reality systems typically provide only visual and audio feedback. The impact of richer, multi-sensory feedback on joint decision-making tasks in VR is still an open area of research. Two independent studies exploring this topic are presented in this paper. We implemented a multi-sensory system that delivers vision, audio, tactile, and smell feedback, and we compared the system to a typical VR system. The scenario placed two users in a virtual theme-park safari ride with a number of non-player character (NPC) passengers to simulate realistic scenarios compared to the real-world and we varied the type and complexity of NPCs reactions to participants. In Experiment 1, we provided both users with either multi-sensory or typical sensory feedback symmetrically as a between-subjects factor, and used NPC reaction type as a within-subjects factor. In Experiment 2, we provided sensory feedback asymmetrically to each user (i.e., one had multi-sensory cues and the other had typical sensory cues) as a between-subjects factor, and used NPC reaction type as a within-subjects factor. We found that the number of sensory channels and NPC reactions did not influence user perception significantly under either symmetric or asymmetric sensory feedback conditions. However, after accounting for individual personality traits (e.g., assertive, passive), as well as any existing relationship between the pairs, we found that increasing the number of sensory channels can significantly improve subjective responses.

This work addresses cybersickness, a major barrier to successful long-exposure immersive virtual reality (VR) experiences since user discomfort frequently leads to prematurely ending such experiences. Starting from sensory conflict theory, we posit that if a vibrating floor delivers vestibular stimuli that minimally match the vibration characteristics of a scenario, the size of the conflict between the visual and vestibular senses will be reduced and, thus, the incidence and/or severity of cybersickness will also be reduced. We integrated a custom-built, computer-controlled vibrating floor in our VR system. To evaluate the system, we implemented a realistic off-road vehicle driving simulator in which participants rode multiple laps as passengers on an off-road course. We programmed the floor to generate vertical vibrations similar to those experienced in real off-road vehicle travel. The scenario and driving conditions were designed to be cybersickness-inducing for users in both the Vibration and No-vibration conditions. We collected subjective and objective data for variables previously shown to be related to levels of cybersickness or presence. These included presence and simulator sickness questionnaires (SSQ), self-rated discomfort levels, and the physiological signals of heart rate, galvanic skin response (GSR), and pupil size. Comparing data between participants in the Vibration group (N=11) to the No-Vibration group (N=11), we found that Delta-SSQ Oculomotor response and the GSR physiological signal, both known to be positively correlated with cybersickness, were significantly lower (with large effect sizes) for the Vibration group. Other variables differed between groups in the same direction, but with trivial or small effect sizes. The results indicate that the floor vibration significantly reduced some measures of cybersickness.

With the continuing development of affordable immersive virtual reality (VR) systems, there is now a growing market for consumer content. The current form of consumer systems is not dissimilar to the lab-based VR systems of the past 30 years: the primary input mechanism is a head-tracked display and one or two tracked hands with buttons and joysticks on hand-held controllers. Over those 30 years, a very diverse academic literature has emerged that covers design and ergonomics of 3D user interfaces (3DUIs). However, the growing consumer market has engaged a very broad range of creatives that have built a very diverse set of designs. Sometimes these designs adopt findings from the academic literature, but other times they experiment with completely novel or counter-intuitive mechanisms. In this paper and its online adjunct, we report on novel 3DUI design patterns that are interesting from both design and research perspectives: they are highly novel, potentially broadly re-usable and/or suggest interesting avenues for evaluation. The supplemental material, which is a living document, is a crowd-sourced repository of interesting patterns. This paper is a curated snapshot of those patterns that were considered to be the most fruitful for further elaboration.

Telepresence robots allow people to participate in remote spaces, yet they can be difficult to manoeuvre with people and obstacles around. We designed a haptic-feedback system called "FeetBack," which users place their feet in when driving a telepresence robot. When the robot approaches people or obstacles, haptic proximity and collision feedback are provided on the respective sides of the feet, helping inform users about events that are hard to notice through the robot's camera views. We conducted two studies: one to explore the usage of FeetBack in virtual environments, another focused on real environments. We found that FeetBack can increase spatial presence in simple virtual environments. Users valued the feedback to adjust their behaviour in both types of environments, though it was sometimes too frequent or unneeded for certain situations after a period of time. These results point to the value of foot-based haptic feedback for telepresence robot systems, while also the need to design context-sensitive haptic feedback.

Supporting perceptual-cognitive tasks is an important part of our daily lives. We use rich, multi-sensory feedback through sight, sound, touch, smell, and taste to support better perceptual-cognitive things we do, such as sports, cooking, and searching for a location, and to increase our confidence in performing those tasks in daily life. Same with real life, the demand for perceptual-cognitive tasks exists in serious VR simulations such as surgical or safety training systems. However, in contrast to real life, VR simulations are typically limited to visual and auditory cues, while sometimes adding simple tactile feedback. This could make it difficult to make confident decisions in VR.In this paper, we investigate the effects of multi-sensory stimuli, namely visuals, audio, two types of tactile (floor vibration and wind), and smell in terms of the confidence levels on a location-matching task which requires a combination of perceptual and cognitive work inside a virtual environment. We also measured the level of presence when participants visited virtual places with different combinations of sensory feedback. Our results show that our multi-sensory VR system was superior to a typical VR system (vision and audio) in terms of the sense of presence and user preference. However, the subjective confidence levels were higher in the typical VR system.

We have explored the effects of sharing manipulated heart rate feedback in collaborative virtual environments. In our study, we created two types of different virtual environments (active and passive) with different levels of interactions and provided three levels of manipulated heart rate feedback (decreased, unchanged, and increased). We measured the effects of manipulated feedback on Social Presence, affect, physical heart rate, and overall experience. We noticed a significant effect of the manipulated heart rate feedback in affecting scariness and nervousness. The perception of the collaborator's valance and arousal was also affected where increased heart rate feedback perceived as a higher valance and lower arousal. Increased heart rate feedback decreased the real heart rate. The type of virtual environments had a significant effect on social presence, heart rate, and affect where the active environment had better performances across these measurements. We discuss the implications of this and directions for future research.

Redirected walking (RDW) techniques provide a way to explore a virtual space that is larger than the available physical space by imperceptibly manipulating the virtual world view or motions. These manipulations may introduce conflicts between real and virtual cues (e.g., visual-vestibular conflicts), which can be disturbing when detectable by users. The empirically established detection thresholds of rotation manipulation for RDW still require a large physical tracking space and are therefore impractical for general-purpose Virtual Reality (VR) applications. We investigate Redirected Jumping (RDJ) as a new locomotion metaphor for redirection to partially address this limitation, and because jumping is a common interaction for environments like games. We investigated the detection rates for different curvature gains during RDJ. The probability of users detecting RDJ appears substantially lower than that of RDW, meaning designers can get away with greater manipulations with RDJ than with RDW. We postulate that the substantial vertical (up/down) movement present when jumping introduces increased vestibular noise compared to normal walking, thereby supporting greater rotational manipulations. Our study suggests that the potential combination of metaphors (e.g., walking and jumping) could further reduce the required physical space for locomotion in VR. We also summarize some differences in user jumping approaches and provide motion sickness measures in our study.

The decisions made by an Air Attack Supervisor (AAS) helicopter co-pilots in aerial firefighting have critical and immediate impacts. It is difficult to always make fast, high quality decisions due to the mental and physical stress being experienced. Real world training exercises have limitations such as safety, cost, time and difficulty in reproducing events, making frequent training infeasible. Virtual Reality (VR) offers new training opportunities, but it is challenging to create a virtual environment with the analogous level of stress experienced in the real-world. In this paper, we investigate the use of a multi-user, collaborative, multi-sensory (vision, audio, tactile) VR system to produce a realistic training environment for practising aerial firefighting training scenarios. We focus on a comparison between our VR training system, an equivalent real-world field training and an existing radio-only exercise currently in use, where we compare Heart-Rate Variability (HRV) and self reported stress using the Short Stress State Questionnaire (SSSQ). We conducted the study with real trainee AAS firefighters to determine the effectiveness of the system. Our results show that there were no significant differences between the VR training exercise and the real-world exercise in terms of the level of stress, measured by HRV, and no significant difference between VR and radio-only exercises, as reported by the SSSQ.

We present a study of the usability and induced workload of a new head-mounted display (HMD) which allows the wearer to more easily switch between tasks in the virtual and real world using multiple optical channels dynamically at the press of a button. Most HMDs provide full immersion by occluding the real world from the view of the user, and replacing it with a computer-generated virtual environment. One trade-off, however, is that such HMDs make it difficult to interact with real-world objects and people, or to react to events while being immersed. Tasks that are simple in the unmediated real world, such as taking a drink or responding to a text, become nearly impossible while wearing an occlusive HMD, and normally require the user to take off the HMD. To address this limitation, we propose an HMD which provides optical channels to view the near-field real world in the periphery of the HMD while still wearing the HMD by pressing a button. We call our approach Multi-channel Dynamic Immersion (MDI). We embedded transparency controllable LCD panels around the periphery of an HMD and installed a fish-eye lens in front of its forward-facing camera. We then compared MDI to three other methods of switching from working in VR to real world activities: (a) a front camera video-see-through method, (b) using dynamic LCDs only, and (c) taking off the HMD. Participants had to interrupt their work in a VR office to carry out typical real world daily activities, such as picking up a mug, responding to a phone call, or replying to a text message. Our results show no significant differences in task execution time between the four conditions. However, we observed a tendency favouring MDI as the easiest to use among the HMDs. In addition, we found a significantly higher rated ease of use for the two HMDs with controllable LCDs compared to the two occlusive HMDs. Our data also underscores that a substantial amount of time is lost when switching between work contexts in occlusive HMDs and the real world.

This study explores ways to increase comfort in Virtual Reality by minimizing cybersickness. Cybersickness is related to classical motion sickness and causes unwanted symptoms when using immersive technologies. We developed a dial interface to accurately capture momentary user cybersickness and feed this information back to the user. Using a seated VR roller coaster environment, we found that the dial is significantly positively correlated with post-immersion questionnaires and is a valid tool compared to verbal rating approaches.

A key challenge to effective storytelling using Virtual Reality (VR), such as with 360-degree videos, is how to direct user attention to important content without taking away user agency for free exploration. In this paper, we introduce the notion of an Action Unit system, composed of social cues such as head and arm movements, as a way of directing users to focus on content important for the given narrative. We applied this idea to a 360-degree VR tour, and evaluated its effects on memory, engagement, enjoyment, and cyber-sickness. The results indicate that the levels of engagement and enjoyment increased when these Action Units were applied. Users also preferred the Action Units for their diegetic aspects.

To interact as fully-tracked avatars with rich hand gestures in Virtual Reality (VR), we often need to wear a tracking suit or attach extra sensors on our bodies. User experience and performance may be impacted by the cumbersome devices and low fidelity behavior representations, especially in social scenarios where good communication is required. In this paper, we use multiple depth sensors and focus on increasing the behavioral fidelity of a participant’s virtual body representation. To investigate the impact of the depth-sensor-based avatar system (full-body tracking with hand gestures), we compared it against a controller-based avatar system (partial-body tracking with limited hand gestures). We designed a VR interview simulation for a single user to measure the effects on presence, virtual body ownership, workload, usability, and perceived self-performance. Specifically, the interview process was recorded in VR, together with all the verbal and non-verbal cues. Subjects then took a third-person view to evaluate their previous performance. Our results show that the depth-sensor-based avatar control system increased virtual body ownership and also improved the user experience. In addition, users rated their non-verbal behavior performance higher in the full-body depth-sensor-based avatar system.

We present Mini-Me, an adaptive avatar for enhancing Mixed Reality (MR) remote collaboration between a local Augmented Reality (AR) user and a remote Virtual Reality (VR) user. The Mini-Me avatar represents the VR user's gaze direction and body gestures while it transforms in size and orientation to stay within the AR user's field of view. A user study was conducted to evaluate Mini-Me in two collaborative scenarios: an asymmetric remote expert in VR assisting a local worker in AR, and a symmetric collaboration in urban planning. We found that the presence of the Mini-Me significantly improved Social Presence and the overall experience of MR collaboration.

Collaboration is an important application area for virtual reality (VR). However, unlike in the real world, collaboration in VR misses important empathetic cues that can make collaborators aware of each other’s emotional states. Providing physiological feedback, such as heart rate or respiration rate, to users in VR has been shown to create a positive impact in single user environments. In this paper, through a rigorous mixed-factorial user experiment, we evaluated how providing heart rate feedback to collaborators influences their collaboration in three different environments requiring different kinds of collaboration. We have found that when provided with real-time heart rate feedback participants felt the presence of the collaborator more and felt that they understood their collaborator’s emotional state more. Heart rate feedback also made participants feel more dominant when performing the task. We discuss the implication of this research for collaborative VR environments, provide design guidelines, and directions for future research.

Situation Awareness (SA) is an essential skill in Air Attack Supervision (AAS) for aerial based wildfire firefighting. The display types used for Virtual Reality Training Systems (VRTS) afford different visual SA depending on the Field of View (FoV) as well as the sense of presence users can obtain in the virtual environment. We conducted a study with 36 participants to evaluate SA acquisition in three display types: a high-definition TV (HDTV), an Oculus Rift Head-Mounted Display (HMD) and a 270° cylindrical simulation projection display called the SimPit. We found a significant difference between the HMD and the HDTV, as well as with the SimPit and the HDTV for the three levels of SA.

Previous work on combining Virtual Reality (VR) and visual cue-exposure therapy has shown promising results that suggest its potential as a tool to support treatments for eating disorders. Visual food cues can elicit cravings in people no matter where they get their exposure from (e.g., photograph, real world, or virtual world). However, there is little work on the influence of olfactory stimuli in VR. Consequently, we investigated the effects of olfactory stimuli and VR on food cravings. In particular, we examined the hypothesis that olfactory and interaction with food in VR can further increase food cravings. The results of this study show that VR can elicit similar effects as exposure to traditional cues when compared to a neutral baseline. Furthermore, the added olfactory cues increased food cravings and the urge to eat the presented food, which also increased when participants were allowed to interact with the virtual food. In conclusion, VR was shown to have considerable potential to be a valid alternative to traditional food-exposure interventions.

To examine the effects of multi-sensory cues during non-fatiguing walking in immersive virtual environments, we selected sensory cues including movement wind, directional wind, footstep vibration, and footstep sounds, and investigated their influence and interaction with each other. We developed a virtual reality system with non-fatiguing walking interaction and low-latency, multi-sensory feedback, and used it to conduct two successive experiments measuring user experience and performance through a triangle-completion task. We noticed some positive effects due to the addition of footstep vibration on task performance, and saw significant improvement in reported user experience due to the added wind and vibration cues.

In this poster, we introduce the notion of using LCD panels for controlling the level of immersion provided by head-mounted displays (HMDs). We propose replacing the cowling around typical “ski-goggle” type HMDs with LCD panels of the type used in commodity active-stereo glasses used in movie theaters. The panels can be controlled using very simple stand-alone circuitry and/or a micro-controller to vary the amount of the real world that is visible in the periphery of the user. This can drastically increase the usability of consumer HMDs, because it allows users to see objects in their immediate surroundings (e.g., the keyboard and mouse), can be used to counter cybersickness by providing natural cues when needed, and introduces no added latency into the system. We show several prototypes of our approach using Google Cardboard, and provide some initial thoughts on the systems.

In virtual reality, hybrid virtual environment (HVE) systems provide the immersed user with multiple interactive representations of the virtual world, and can be effectively used for 3D interaction tasks with highly diverse requirements. We present a new HVE metaphor called Object Impersonation that allows the user to not only manipulate a virtual object from outside, but also become the object, and maneuver from inside. This approach blurs the line between travel and object manipulation, leading to efficient cross-task interaction in various task scenarios. Using a tablet- and HMD-based HVE system, two different designs of Object Impersonation were implemented, and compared to a traditional, non-hybrid 3D interface for three different object manipulation tasks. Results indicate improved task performance and enhanced user experience with the added orientation control from the object's point of view. However, they also revealed higher cognitive overhead to attend to both interaction contexts, especially without sufficient reference cues in the virtual environment.

We report on two experiments that deploy low-frequency audio and strong vibrations to induce haptic-like sensations throughout the human body. Vibration is quite frequently deployed in immersive systems, for example to provide collision feedback, but its actual effects are not well understood [Kruijff & Pander 2005; Kruijff et al. 2015]. The starting point of our experiments was a study by Rasmussen [Rasmussen 1982], which found that different vibration frequencies were experienced differently throughout the body. We will show how vibrations affect sensations throughout the body and may provide some directional cues to some parts of the body, yet also illustrate the difficulties.

The IEEE International Symposium on Mixed and Augmented Reality (ISMAR) is the leading venue for publishing the latest Mixed and Augmented Reality research, applications, and technologies. This special section presents significantly extended versions of the five best papers from the IEEE ISMAR 2014 proceedings. Within the past few years, Augmented Reality (AR) has reached a critical mass in both research and commercial applications. It is now becoming truly feasible to use augmented reality to place graphics anywhere at any time. However, although the basic capabilities exist, many open research problems continue. This collection of papers considers underlying issues and technologies. IEEE ISMAR 2014 had 89 paper submissions; each paper was reviewed by at least four experts in the field. An international programcommittee of 15 ARexperts invited reviewers, led discussions, invited a rebuttal by the paper authors and prepared a consensus review. To select the final papers for publication, an online two-day PC meeting was held connecting three continents, where each paper was discussed. Thirty-five papers were accepted either as long or short publications, giving an overall acceptance rate of 40%. An independent Award Committee reviewed the highest- ranked submissions again to determine the awards for Best Paper and Honorable Mention. For this special section, the authors of the award papers were invited to submit an extended version of their conference papers, with a clear focus on additional content that expands the scientific contribution of the original conference paper. A standard TVCG reviewing cycle was initiated in which all papers were reviewed, feedback was provided, and papers were revised to suit. Out of all submitted papers, less than 6% appear in this TVCG Special Section.

Supervision and teleoperation of high degree-of-freedom robots is a complex task due to environmental constraints such as obstacles and limited communication, as well as task specific requirements such as using more than one end-effector at the same time. In this work we present a supervision and teleoperation framework that allows an operator to see the surroundings of a robot in 3D, make necessary adjustments for a dual or single arm manipulation task, preview the task in simulation before execution, and finally execute the task on a real robot. The framework has been applied to the valve turning task of the DARPA Robotics Challenge on the PR2, Hubo2+, and DRCHubo robots.

In this paper, we report lessons learned through the design of a framework for teleoperating a humanoid robot to perform a manipulation task. We present a software framework for cooperative traded control that enables a team of operators to control a remote humanoid robot over an unreliable communications link. The framework produces statically-stable motion trajectories that are collision-free and respect end-effector pose constraints. After operator confirmation, these trajectories are sent over the data link for execution on the robot. Additionally, we have defined a clear operational procedure for the operators to manage the teleoperation task. We applied our system to the valve turning task in the DARPA Robotics Challenge (DRC). Our framework is able to perform reliably and is resilient to unreliable network conditions, as we demonstrate in a set of test runs performed remotely over the internet. We analyze our approach and discuss lessons learned which may be useful for others when designing such a system.

In this paper we revisit the design of leaning-based travel interfaces and propose a design space to categorize existing implementations. Within the design space, frontal and sidewise stances when using a flying surfboard interface were compared through a user study. The interfaces were adapted and improved from our previous designs using a body-mounted, multi-touch touchpad. Two different experiments were designed and conducted that focus on user performance and virtual world cognition, respectively. The results suggest better user performance and user experience when using the frontal stance, although no better spatial orientation or virtual world cognition was identified. Further, user interviews revealed that despite the realistic simulation of skateboarding/snowboarding, the sidewise stance suffers from poor usability due to inefficient and inaccurate turning control and confusion between the viewing and movement directions. Based on these results, several guidelines are proposed to aid the design of leaning-based travel interfaces for immersive virtual reality applications.

Welcome to the 18th IEEE Virtual Reality conference. As with past conferences, we hope to excite and inspire you with the many events scheduled as part of IEEE VR 2011. Also, we want to emphasize that IEEE VR comes to Asia for only the second time in its long history. It was in Yokohama, Japan in 2001, and this time it will be held in Singapore, one of the most advanced and still rapidly advancing countries in Asia.

The four papers in this special section are expanded versions of the four best papers from the IEEE Virtual Reality (VR) Proceedings.

This paper presents a markerless tracking technique targeted to the Windows Mobile Pocket PC platform. The primary aim of this work is to allow the development of standalone augmented reality applications for handheld devices based on natural feature tracking of fully 3-Dimensional objects. In order to achieve this goal, a model-based tracking approach that relies on edge information was adopted. Since it does not require high processing power, it is suitable for constrained devices such as handhelds. The OpenGL ES graphics library was used to detect the visible edges in a given frame, taking advantage of graphics hardware acceleration when available. In addition, a subset of two computer vision libraries was ported to the Pocket PC platform in order to provide some required algorithms to the markerless mobile solution. They were also adapted to use fixed-point math, with the purpose of improving the overall performance of the routines. The port of these libraries opens up the possibility of having other computer-vision tasks being executed on mobile platforms. An augmented reality application was created using the implemented technique and evaluations were done regarding tracking performance, accuracy and robustness. In most of the tests, the frame rates obtained are suitable for handheld augmented reality and a reasonable estimation of the object pose was provided.

Much of Virtual Reality (VR) is about creating virtual worlds that are believable. But though the visual and audio experiences we provide today technically approach the limits of human sensory systems, there is still something lacking; something beyond sensory fidelity hinders us from fully buying into the worlds we experience through VR technology.

Augmented reality (AR) is the mixing of computer-generated stimuli with real-world stimuli. In this paper, we present results from a controlled, empirical study comparing three ways of delivering spatialized audio for AR applications: a speaker array, headphones, and a bone-conduction headset. Analogous to optical-see-through AR in the visual domain, hear-through AR allows users to receive computer-generated audio using the bone-conduction headset, and real-world audio using their unoccluded ears. Our results show that subjects achieved the best accuracy using a speaker array physically located around the listener when stationary sounds were played, but that there was no difference in accuracy between the speaker array and the bone-conduction device for sounds that were moving, and that both devices outperformed standard headphones for moving sounds. Subjective comments by subjects following the experiment support this performance data.

We present a novel approach for mixing real and computer-generated audio for augmented reality (AR) applications. Analogous to optical-see-through and video-see-through techniques in the visual domain, we present Hear-Through and Mic-Through audio AR. Hear-Through AR uses a bone-conduction headset to deliver computer-generated audio, while leaving the ear canals free to receive audio from the surrounding environment. Mic-Through AR allows audio signals captured from ear-worn microphones to be mixed with computer-generated audio in the computer, and delivered to the user over headphones. We present preliminary results from an empirical user study conducted to compare a bone-conduction device, headphones, and a speaker array. The results show that subjects achieved the best accuracy using an array of speakers physically located around the listener when stationary sounds were played, but that there was no difference in accuracy between the speaker array and the bone-conduction device for sounds that were moving, and that both devices outperformed standard headphones for moving sounds.

We present a new classification framework for describing augmented reality (AR) applications based on where the mixing of real and computer-generated stimuli takes place. In addition to "classical " visual AR techniques, such as optical-see-through and video-see-through AR, our framework encompasses AR directed at the other senses as well. This "axis of mixing location " is a continuum ranging from the physical environment to the human brain. There are advantages and disadvantages of mixing at different points along the continuum, and while there is no "best" location, we present sample usage scenarios that illustrate the expressiveness of this classification approach.

This paper presents empirical results to support the use of vibrotactile cues as a means of improving user performance on a spatial task. In a building-clearing exercise, directional vibrotactile cues were employed to alert subjects to areas of the building that they had not yet cleared, but were currently exposed to. Compared with performing the task without vibrotactile cues, subjects were exposed to uncleared areas a smaller percentage of time, and cleared more of the overall space, when given the added vibrotactile stimulus. The average length of each exposure was also significantly less when vibrotactile cues were present.

We present recent results we have obtained in the design of a wireless tactor system for use in virtual as well as real environments. We describe a system we have designed that uses vibration motors wired to a body-worn control box, which in turn communicates with a host computer using a wireless connection. Though we have successfully integrated this system into an existing VR-based dismounted infantry simulator, it is clear that expanding the body coverage and number of vibration points requires a solution that is completely wireless. To this end, we describe a working prototype we have built that, though still somewhat large and feature-limited, has convinced us of the merit of our approach.

This paper presents work we have done on the design and implementation of an untethered system to deliver haptic cues for use in immersive virtual environments through a body-worn garment. Our system can control a large number of body-worn vibration units, each with individually controllable vibration intensity. Several design iterations have helped us to refine the system and improve such aspects as robustness, ease of donning and doffing, weight, power consumption, cable management, and support for many different types of feedback units, such as pager motors, solenoids, and muffin fans. In addition, experience integrating the system into an advanced virtual reality system has helped define some of the design constraints for creating wearable solutions, and to further refine our implementation.

This paper presents results from two experiments into the use of vibrotactile cues for near-field haptics in virtual environments. In one experiment, subjects were tested on their ability to identify the location of a one-second vibrotactile stimulus presented to a single tactor of a 3-by-3 array on their back. We recorded an 84% correct identification rate. In a second experiment, subjects were asked to match the intensity of a vibrotactile stimulus presented at one location with the intensity at another location. We found that subjects could match the intensities to within 7Hz if the reference and adjustable stimuli were presented at the same location, but only to within 18Hz otherwise.

The paper presents a portable system and method for recognizing the 26 hand shapes of the American Sign Language alphabet, using a novel glove-like device. Two additional signs, 'space', and 'enter' are added to the alphabet to allow the user to form words or phrases and send them to a speech synthesizer. Since the hand shape for a letter varies from one signer to another, this is a 28-class pattern recognition system. A three-level hierarchical classifier divides the problem into "dispatchers" and "recognizers." After reducing pattern dimension from ten to three, the projection of class distributions onto horizontal planes makes it possible to apply simple linear discrimination in 2D, and Bayes' Rule in those cases where classes had features with overlapped distributions. Twenty-one out of 26 letters were recognized with 100% accuracy; the worst case, letter U, achieved 78%.

The paper reports empirical results from two studies of effective user interaction in immersive virtual environments. The use of 2D interaction techniques in 3D environments has received increased attention recently. We introduce two new concepts to the previous techniques: the use of 3D widget representations; and the imposition of simulated surface constraints. The studies were identical in terms of treatments, but differed in the tasks performed by subjects. In both studies, we compared the use of two-dimensional (2D) versus three-dimensional (3D) interface widget representations, as well as the effect of imposing simulated surface constraints on precise manipulation tasks. The first study entailed a drag-and-drop task, while the second study looked at a slider-bar task. We empirically show that using 3D widget representations can have mixed results on user performance. Furthermore, we show that simulated surface constraints can improve user performance on typical interaction tasks in the absence of a physical manipulation surface. Finally, based on these results, we make some recommendations to aid interface designers in constructing effective interfaces for virtual environments.

This paper reports empirical results from a study into the use of 2D widgets in 3D immersive virtual environments. Several researchers have proposed the use of 2D interaction techniques in 3D environments, however little empirical work has been done to test the usability of such approaches. We present the results of two experiments conducted on low-level 2D manipulation tasks within an immersive virtual environment. We empirically show that the addition of passive-haptic feedback for use in precise UI manipulation tasks can significantly increase user performance. Furthermore, users prefer interfaces that provide a physical surface, and that allow them to work with interface widgets in the same visual field of view as the objects they are modifying.

The study of human-computer interaction within immersive virtual environments requires us to balance what we have learned from the design and use of desktop interfaces with novel approaches to allow us to work effectively in three dimensions. While some researchers have called for revolutionary interfaces for these new environments, devoid of two-dimensional (2D) desktop widgets, others have taken a more evolutionary approach. Windowing within immersive virtual environments is an attempt to apply 2D interface techniques to three-dimensional (3D) worlds. 2D techniques are attractive because of their proven acceptance and widespread use on the desktop. With current methods environments, however, it is difficult for users of 3D worlds to perform precise manipulations, such as dragging sliders, or precisely positioning or orienting objects. We have developed a testbed designed to take advantage of bimanual interaction, proprioception, and passive-haptic feedback. We present preliminary results from an empirical study of 2D interaction in 3D environments using this system. We use a window registered with a tracked, physical surface, to provide support for precise manipulation of interface widgets displayed in the virtual environment.