Advancements in wearable technologies have made the use of physiological signals, such as Electrodermal Activity (EDA) and Heart Rate Variability (HRV), more prevalent for detecting changes in the autonomic nervous system within virtual reality (VR). However, the challenge lies in utilizing these signals to objectively detect presence in VR, which typically relies on self-reports that can be inherently biased. This paper addresses this issue and presents a study ( N =26) that investigates the effect that different levels of presence has on physiological responses in VR. A neutral VR environment was created that incorporated three levels of presence (high, medium and low) that were invoked by tuning different parameters. Participants wore a wrist-worn wearable device that captured their physiological signals whilst they experienced each of these environments. Results indicated that tonic and phasic components of the EDA signal were significant in differentiating between the levels. Two novel features, constructed using both the phasic and tonic components of EDA, successfully differentiated between presence levels. Analysis of the HRV data illustrated a significant difference between the low and medium levels using the ratio between low frequency to high frequency. • Validation of the design of three different levels of presence in a VR environment. • Development of two novel features (cf1, cf2) significantly differentiated between levels of presence. • Poincare maps illustrate the variability in the data between different presence levels.

Researchers have used machine learning approaches to identify motion sickness in VR experience. These approaches would certainly benefit from an accurately labeled, real-world, diverse dataset that enables the development of generalizable ML models. We introduce 'VR.net', a dataset comprising 165-hour gameplay videos from 100 real-world games spanning ten diverse genres, evaluated by 500 participants. VR.net accurately assigns 24 motion sickness-related labels for each video frame, such as camera/object movement, depth of field, and motion flow. Building such a dataset is challenging since manual labeling would require an infeasible amount of time. Instead, we implement a tool to automatically and precisely extract ground truth data from 3D engines' rendering pipelines without accessing VR games' source code. We illustrate the utility of VR.net through several applications, such as risk factor detection and sickness level prediction. We believe that the scale, accuracy, and diversity of VR.net can offer unparalleled opportunities for VR motion sickness research and beyond.We also provide access to our data collection tool, enabling researchers to contribute to the expansion of VR.net.

Training systems are used in many industries, ranging from surgery to space missions to rehabilitation. Virtual Reality (VR) is a technology that has been incorporated as an effective tool in such training systems to simulate the environment, especially in situations where the training can’t take place in the actual environment. For a training environment and task to be effective, it must sufficiently challenge the trainee. One parameter that can be used to measure this is cognitive load (CL), which is defined as the amount of working memory used while performing a learning task. This parameter needs to be sufficiently high to maximize learning but not too high as to overload the trainee. However, the challenge is to detect this state using objective physiological measures, which can be collected during the entire task. This paper presents a study to classify CL using a combination of Electroencephalogram (EEG) and Electrodermal Activity (EDA) signals during a procedural VR training task. Thirty participants undertook a study where they built a designated model within a given time over multiple levels that were constructed to induce low to high CL. Features generated from the data were subject to feature selection (FS), which was undertaken using the Mutual Information (MI) technique. Binary classification models were developed using Support Vector Machines (SVM), Random Forest (RF), k-Nearest Neighbors (kNN), Extreme Gradient Boosting (Xgboost) and Multi-Layer Perceptrons (MLP). Results illustrated that the Xgboost classifier performed the best with an F1-score of $0.831 \pm 0.030$ and accuracy of $0.805 \pm 0.033.$ SHAP analysis of the features illustrated greater contributions from the frontal and occipital regions of the brain and frequency domain features from tonic skin conductance.

This paper presents a novel solution for estimating simulator sickness in HMDs using machine learning and 3D motion data, informed by user-labeled simulator sickness data and user analysis. We conducted a novel VR user study, which decomposed motion data and used an instant dial-based sickness scoring mechanism. We were able to emulate typical VR usage and collect user simulator sickness scores. Our user analysis shows that translation and rotation differently impact user simulator sickness in HMDs. In addition, users’ demographic information and self-assessed simulator sickness susceptibility data are collected and show some indication of potential simulator sickness. Guided by the findings from the user study, we developed a novel deep learning-based solution to better estimate simulator sickness with decomposed 3D motion features and user profile information. The model was trained and tested using the 3D motion dataset with user-labeled simulator sickness and profiles collected from the user study. The results show higher estimation accuracy when using the 3D motion data compared with methods based on optical flow extracted from the recorded video, as well as improved accuracy when decomposing the motion data and incorporating user profile information.

Presence, the feeling of being there, is an important factor that affects the overall experience of Virtual Reality (VR). Higher presence commonly provides a better experience in VR than lower presence. However, presence is commonly measured subjectively through postexperience questionnaires, which can suffer from participant biases, dishonest answers, and fatigue. It can also be difficult for subjects to accurately remember their feelings of presence after they have left the VR experience. In this paper, we measured the effects of different levels of presence (high and low) in VR using physiological and neurological signals. The experiment involved 24 participants in a between-subjects design. Results indicated a significant effect of presence on both physiological and neurological signals. We noticed that higher presence results in higher heart rate, less visual stress, higher theta and beta activities in the frontal region, and higher alpha activities in the parietal region. These findings and insights could lead to an alternative objective measure of presence.

We have explored the effects of sharing manipulated heart rate feedback in collaborative virtual environments. In our study, we created two types of different virtual environments (active and passive) with different levels of interactions and provided three levels of manipulated heart rate feedback (decreased, unchanged, and increased). We measured the effects of manipulated feedback on Social Presence, affect, physical heart rate, and overall experience. We noticed a significant effect of the manipulated heart rate feedback in affecting scariness and nervousness. The perception of the collaborator's valance and arousal was also affected where increased heart rate feedback perceived as a higher valance and lower arousal. Increased heart rate feedback decreased the real heart rate. The type of virtual environments had a significant effect on social presence, heart rate, and affect where the active environment had better performances across these measurements. We discuss the implications of this and directions for future research.

Virtual Reality (VR) is effective in various training scenarios across multiple domains, such as education, health and defense. However, most of those applications are not adaptive to the real-time cognitive or subjectively experienced load placed on the trainee. In this paper, we explore a cognitively adaptive training system based on real-time measurement of task related alpha activity in the brain. This measurement was made by a 32-channel mobile Electroencephalography (EEG) system, and was used to adapt the task difficulty to an ideal level which challenged our participants, and thus theoretically induces the best level of performance gains as a result of training. Our system required participants to select target objects in VR and the complexity of the task adapted to the alpha activity in the brain. A total of 14 participants undertook our training and completed 20 levels of increasing complexity. Our study identified significant differences in brain activity in response to increasing levels of task complexity, but response time did not alter as a function of task difficulty. Collectively, we interpret this to indicate the brain's ability to compensate for higher task load without affecting behaviourally measured visuomotor performance.

In this work we present a redirected walking scheme suitable for shared spaces in a virtual reality environment. We show our redirected walking to work for the case of two physical spaces (a host and a guest) being merged into a single virtual host space. The redirection is based on warping the guest space into the host space using a conformal mapping that preserves local shape and features.

Collaboration is an important application area for virtual reality (VR). However, unlike in the real world, collaboration in VR misses important empathetic cues that can make collaborators aware of each other’s emotional states. Providing physiological feedback, such as heart rate or respiration rate, to users in VR has been shown to create a positive impact in single user environments. In this paper, through a rigorous mixed-factorial user experiment, we evaluated how providing heart rate feedback to collaborators influences their collaboration in three different environments requiring different kinds of collaboration. We have found that when provided with real-time heart rate feedback participants felt the presence of the collaborator more and felt that they understood their collaborator’s emotional state more. Heart rate feedback also made participants feel more dominant when performing the task. We discuss the implication of this research for collaborative VR environments, provide design guidelines, and directions for future research.

Interfaces for collaborative tasks, such as multiplayer games can enable more effective and enjoyable collaboration. However, in these systems, the emotional states of the users are often not communicated properly due to their remoteness from one another. In this paper, we investigate the effects of showing emotional states of one collaborator to the other during an immersive Virtual Reality (VR) gameplay experience. We created two collaborative immersive VR games that display the real-time heart-rate of one player to the other. The two different games elicited different emotions, one joyous and the other scary. We tested the effects of visualizing heart-rate feedback in comparison with conditions where such a feedback was absent. The games had significant main effects on the overall emotional experience.

To examine the effects of multi-sensory cues during non-fatiguing walking in immersive virtual environments, we selected sensory cues including movement wind, directional wind, footstep vibration, and footstep sounds, and investigated their influence and interaction with each other. We developed a virtual reality system with non-fatiguing walking interaction and low-latency, multi-sensory feedback, and used it to conduct two successive experiments measuring user experience and performance through a triangle-completion task. We noticed some positive effects due to the addition of footstep vibration on task performance, and saw significant improvement in reported user experience due to the added wind and vibration cues.

Handheld Augmented Reality (HAR) has the potential to introduce Augmented Reality (AR) to large audiences due to the widespread use of suitable handheld devices. However, many of the current HAR systems are not considered very practical and they do not fully answer to the needs of the users. One of the challenging areas in HAR is the in-situ AR content creation where the correct and accurate positioning of virtual objects to the real world is fundamental. Due to the hardware limitations of handheld devices and possible restrictions in the environment, the correct 3D positioning of objects can be difficult to achieve we are unable to use AR markers or correctly map the 3D structure of the environment. We present SlidAR, a 3D positioning for Simultaneous Localization And Mapping (SLAM) based HAR systems. SlidAR utilizes 3D ray-casting and epipolar geometry for virtual object positioning. It does not require a perfect 3D reconstruction of the environment nor any virtual depth cues. We have conducted a user experiment to evaluate the efficiency of SlidAR method against an existing device-centric positioning method that we call HoldAR. Results showed that SlidAR was significantly faster, required significantly less device movement, and also got significantly better subjective evaluation from the test participants. SlidAR also had higher positioning accuracy, although not significantly.

Augmented Reality systems that run interactively and in real time, using high quality graphical displays and sensational cues, can create the illusion of virtual objects appearing to be real. This paper presents the design, implementation, and evaluation of BurnAR, a novel demonstration which enables users to experience the illusion of seeing their own hands burning, which we achieve by overlaying virtual flames and smoke on their hands. Surprisingly, some users reported an involuntary warming sensation of their hands. Based on these comments, we hypothesized that stimulation of multiple sensory modalities presented in this AR environment can induce an involuntary experience in an additional sensory pathway: observation of virtual flames resulting in a heat sensation. This cross-modal transfer, known as virtual synesthe-sia, is a temporary experience which affects some people who are not synesthetes and only lasts for a short time during the illusory experience. To verify our hypothesis, we conducted an exploratory study where participants experienced the BurnAR demonstration under controlled conditions.

Augmented Reality (AR) applications on mobile devices like smartphones and tablet computers have become increasingly popular. In this paper, for the first time in the AR domain, we present: (1) the influence of different handheld displays and (2) the exocentric depth perception. Unlike egocentric depth perception, exocentric depth perception has not been investigated in AR.

Augmented Reality systems that run interactively and in real time, using high quality graphical displays and sensational cues, can create the illusion of virtual objects appearing to be real. This paper presents the design and implementation of BurnAR, a demonstration which enables users to experience the illusion of seeing their own hands burning, which we achieve by overlaying virtual flames and smoke on their hands. Surprisingly, some users reported an involuntary warming sensation of their hands.

In Virtual Environments (VE), users are often facing tasks that involve direct manipulation of virtual objects at close distances, such as touching, grabbing, placement. In immersive systems that employ head-mounted displays these tasks could be quite challenging, due to lack of convergence of virtual cameras.

Reliable and unobtrusive eye tracking remains a technical challenge for immersive virtual environment, especially when Head Mounted Displays (HMD) are used for visualization and users are allowed to move freely in the environment. In this work, we provide experimental evidence that gaze direction can be safely approximated by user head rotation, in HMD-based Virtual Reality (VR) applications, where users actively interact with the environment. We discuss the application range of our approach and consider possible extensions.

The calibration of optical see-through head-mounted displays is an important fundament for correct object alignment in augmented reality. Any calibration process for OSTHMDs requires users to align 2D points in screen space with 3D points in the real world and to confirm each alignment. In this poster, we present the results of our empiric evaluation where we compared four confirmation methods: Keyboard, Hand-held, Voice, and Waiting. The Waiting method, designed to reduce head motion during confirmation, showed a significantly higher accuracy than all other methods. Averaging over a time frame for sampling user input before the time of confirmation improved the accuracy of all methods in addition. We conducted a further expert study proving that the results achieved with a video see-through head-mounted display showed valid for optical see-through head-mounted display calibration, too.

In the past, several systems have been presented that enable users to view occluded points of interest using Augmented Reality X-ray visualizations. It is challenging to design a visualization that provides correct occlusions between occluder and occluded objects while maximizing legibility. We have previously published an Augmented Reality X-ray visualization that renders edges of the occluder region over the occluded region to facilitate correct occlusions while providing foreground context. While this approach is simple and works in a wide range of situations, it provides only minimal context of the occluder object. In this paper, we present the background, design, and implementation of our novel visualization technique that aims at providing users with richer context of the occluder object. While our previous visualization only employed one salient feature (edges) to determine which parts of the occluder to display, our novel visualization technique is an initial attempt to explore the design space of employing multiple salient features for this task. The prototype presented in this paper employs three additional salient features: hue, luminosity, and motion. We have conducted two evaluations with human participants to investigate the benefits and limitations of our prototype compared to our previous system. The first evaluation showed that although our novel visualization provides a richer context of the occluder object, it does not impede users to select objects in the occluded area; but, it also indicated problems in our prototype. In the second evaluation, we have investigated these problems through an online survey with systematically varied occluder and occluded scenes, focussing on the qualitative aspects of our visualizations. The results were encouraging, but pointed out that our novel visualization needs a higher level of adaptiveness.

Most of today's mobile internet devices contain facilities to display maps of the user's surroundings with points of interest embedded into the map. Other researchers have already explored complementary, egocentric visualizations of these points of interest using mobile mixed reality. Being able to perceive the point of interest in detail within the user's current context is desirable, however, it is challenging to display off-screen or occluded points of interest. We have designed and implemented space-distorting visualizations to address these situations. While this class of visualizations has been extensively studied in information visualization, we are not aware of any attempts to apply them to augmented or mixed reality. Based on the informal user feedback that we have gathered, we have performed several iterations on our visualizations. We hope that our initial results can inspire other researchers to also investigate space-distorting visualizations for mixed and augmented reality.

Enabling users to accurately perceive the correct depth of occluded objects is one of the major challenges in user interfaces for Mixed Reality (MR). Therefore, several visualization techniques and user evaluations for this area have been published. Our research is fo-cused on photorealistic X-ray type visualizations in outdoor envi-ronments. In this paper, we present an evaluation of depth percep-tion in far-field distances through two photorealistic visualizations of occluded objects (X-ray and Melt) in the presence and absence of a depth cue. Our results show that the distance to occluded ob-jects was underestimated in all tested conditions. This finding is curious, as it contradicts previously published results of other re-searchers. The Melt visualization coupled with a depth cue was the most accurate among all the experimental conditions.

Throughout the last decade, mobile information browsing has become a widely-adopted practice. Most of today's mobile Internet devices contain facilities to display maps of the user's surroundings with points of interest embedded into the map. Other researchers have already explored complementary, egocentric visualizations of these points of interest using mobile mixed reality. However, it is challenging to display off-screen or occluded points of interest. We have designed and implemented space-distorting visualizations to address these situations. Based on the informal user feedback that we have gathered, we have performed several iterations on our visualizations. We hope that our initial results can inspire other researchers to also investigate space-distorting visualizations for mixed and augmented reality.