Virtual reality training systems require the careful design of content presentation, user embodiment, and overall user experience. We explore the impact of different perspectives (first-person and third-person) and virtual self-visualization techniques (VSVTs: mirrors and external avatars) on user embodiment, performance and experience. In a study with 28 participants learning karate movements, we tested four combinations of these factors. Results indicate that perspective influences visual focus and embodiment, while VSVTs affect movement execution, particularly in the third-person avatar condition. Measurements of physiological activity, workload, presence, and enjoyment found no significant overall advantages for any of the conditions. Interviews revealed that most participants preferred the familiar first-person mirror combination, although participants in third-person perspective focused more on their own body and noted the helpfulness of this viewpoint. The study demonstrates that alternative perspectives and visualization techniques offer valuable training options, as these conditions did not produce significant differences in measured cognitive load when compared with each other. Future VR training systems should incorporate interactive feedback and customization options to accommodate individual preferences and optimize learning experiences.

Virtual reality (VR) finds various applications in productivity, entertainment, and training, often requiring substantial working memory and attentional resources. Effective task performance in VR relies on prioritizing relevant information and suppressing distractions through internal attention. However, current VR systems fail to account for the impact of working memory loads, leading to over or under-stimulation. In this work, we designed an adaptive system using Electroencephalography (EEG) correlates of external and internal attention to support working memory tasks. Participants engaged in a visual working memory N-Back task, where we adapted the visual complexity of distracting elements. Our study demonstrated that EEG frontal theta and parietal alpha frequency bands effectively adjust dynamic visual complexity. The adaptive system improved task performance and reduced perceived workload compared to a reverse adaptation. Furthermore, we trained a Linear Discriminant Analysis (LDA) model and achieved a classification accuracy of 79.4% for distinguishing internal and external attention states using EEG frequency features, demonstrating the feasibility of EEG-based models for real-time attention state detection. These results highlight the potential of EEG-based adaptive systems to balance distraction management and maintain user engagement without causing cognitive overload. • Developed a VR system using EEG to balance performance and engagement. • Improved task performance with subtle, natural adaptations to environment factors. • Achieved 79.4% accuracy classifying attention states using an LDA model.

Object detection tasks still often require manual image analysis. Using Machine Learning (ML) instead creates accountability challenges, necessitating experts for model refinement, which is costly and takes time. We investigate integrating crowd knowledge as a cost-effective alternative. While human capabilities in recognizing complex patterns and perceiving variations can still outperform machines and improve an imperfect ML model, ML predictions can compensate for the crowd’s lack of expertise. Our investigation (N=28 non-expert) in oil spill detection shows that adopting an ML-assisted UI elevates precision and recall by over 11% and increases efficiency by 29% compared to a non-assisted UI. Considering agreement among non-expert crowd workers further improved precision by 8% and recall by almost 5%, which is also substantially beyond pure ML performance. Our work contributes an approach for combining crowd knowledge and ML to advance human-AI collaboration in oil spill detection.

Physiologically-adaptive Virtual Reality systems dynamically adjust virtual content based on users’ physiological signals to enhance interaction and achieve specific goals. However, as different users’ cognitive states may underlie multivariate physiological patterns, adaptive systems necessitate a multimodal evaluation to investigate the relationship between input physiological features and target states for efficient user modeling. Here, we investigated a multimodal dataset (EEG, ECG, and EDA) while interacting with two different adaptive systems adjusting the environmental visual complexity based on EDA. Increased visual complexity led to increased alpha power and alpha-theta ratio, reflecting increased mental fatigue and workload. At the same time, EDA exhibited distinct dynamics with increased tonic and phasic components. Integrating multimodal physiological measures for adaptation evaluation enlarges our understanding of the impact of system adaptation on users’ physiology and allows us to account for it and improve adaptive system design and optimization algorithms.

Future VR environments envision adaptive and personalized interactions.To this aim, attention detection in VR settings would allow for diverse applications and improved usability.However, attention-aware VR systems based on EEG data suffer from long training periods, hindering generalizability and widespread adoption.This work addresses the challenge of person-independent, training-free VR BCI classifying internal and external attention in VR.We compared the performance of four classifiers on an EEG dataset (N=24) featuring internal and external attention labeled classes.With the goal of online adaptation, we tested overall accuracy, different window lengths of the data, and training split to optimize the trade-off between window length and classification accuracy.Our results show that models using a complete EEG band combination consistently achieve the highest accuracy, with Linear Discriminant Analysis particularly benefiting from full-band data.The window length impacts most models' performance with short windows.LDA achieved optimal accuracy around 6.3 seconds, SVM and NN around 6.5 and 6 seconds, respectively, and RF reached stability at 6 seconds.Lastly, increasing training data ratios improved accuracy gains consistently across models.We discuss the potential of machine learning to model EEG correlates of internal and external attention as online inputs for adaptive VR systems.

Physical training in virtual environments, such as VR, has gained popularity, especially due to the coronavirus pandemic. VR training offers new opportunities compared to traditional methods, including the use of different perspectives, mirrors, and avatars to enhance the understanding of personal movements. However, the interaction of these elements has been less studied. To address this, we developed VReflect, a VR environment that uses mirrors and avatars as virtual self-visualization techniques (VSVT) to improve self-awareness during movement training. In a preliminary study on learning beginner karate movements, we tested four combinations of perspectives and VSVTs. The results indicate that neither first-person nor third-person perspectives can be universally recommended, which is in alignment with previous work. Interviews revealed a preference for the traditional combination of mirrors and first-person perspective.

While Mixed Reality allows the seamless blending of digital content in users' surroundings, it is unclear if its fusion with physical information impacts users' perceptual and cognitive resources differently. While the fusion of digital and physical objects provides numerous opportunities to present additional information, it also introduces undesirable side effects, such as split attention and increased visual complexity. We conducted a visual search study in three manifestations of mixed reality (Augmented Reality, Augmented Virtuality, Virtual Reality) to understand the effects of the environment on visual search behavior. We conducted a multimodal evaluation measuring Fixation-Related Potentials (FRPs), alongside eye tracking to assess search efficiency, attention allocation, and behavioral measures. Our findings indicate distinct patterns in FRPs and eye-tracking data that reflect varying cognitive demands across environments. Specifically, AR environments were associated with increased workload, as indicated by decreased FRP - P3 amplitudes and more scattered eye movement patterns, impairing users' ability to identify target information efficiently. Participants reported AR as the most demanding and distracting environment. These insights inform design implications for MR adaptive systems, emphasizing the need for interfaces that dynamically respond to user cognitive load based on physiological inputs.

Mixed reality enables users to immerse themselves in high-workload interaction spaces like office work scenarios. We envision physiologically adaptive systems that can move users into different mixed reality manifestations, to improve their focus on the primary task. However, it is unclear which manifestation is most conducive for high productivity and engagement. In this work, we evaluate whether physiological indicators for engagement can be discriminated for different manifestations. For this, we engaged participants in a typing task in three different mixed reality manifestations (augmented reality, augmented virtuality, virtual reality) and monitored physiological correlates (EEG, ECG, and eye tracking) of users' engagement and workload. We found that users achieved best typing performances in augmented reality and augmented virtuality. At the same time, physiological engagement peaked in augmented virtuality, while workload decreased. We conclude that augmented virtuality strikes a good balance between the different manifestations, as it facilitates displaying the physical keyboard for improved typing performance and, at the same time, allows one to block out the real world, removing many real-world distractors.

Mixed Reality allows us to integrate virtual and physical content into users' environments seamlessly. Yet, how this fusion affects perceptual and cognitive resources and our ability to find virtual or physical objects remains uncertain. Displaying virtual and physical information simultaneously might lead to divided attention and increased visual complexity, impacting users' visual processing, performance, and workload. In a visual search task, we asked participants to locate virtual and physical objects in Augmented Reality and Augmented Virtuality to understand the effects on performance. We evaluated search efficiency and attention allocation for virtual and physical objects using event-related potentials, fixation and saccade metrics, and behavioral measures. We found that users were more efficient in identifying objects in Augmented Virtuality, while virtual objects gained saliency in Augmented Virtuality. This suggests that visual fidelity might increase the perceptual load of the scene. Reduced amplitude in distractor positivity ERP, and fixation patterns supported improved distractor suppression and search efficiency in Augmented Virtuality. We discuss design implications for mixed reality adaptive systems based on physiological inputs for interaction.

Social media platforms use short, highly engaging videos to catch users’ attention. While the short-form video feeds popularized by TikTok are rapidly spreading to other platforms, we do not yet understand their impact on cognitive functions. We conducted a between-subjects experiment (N = 60) investigating the impact of engaging with TikTok, Twitter, and YouTube while performing a Prospective Memory task (i.e., executing a previously planned action). The study required participants to remember intentions over interruptions. We found that the TikTok condition significantly degraded the users’ performance in this task. As none of the other conditions (Twitter, YouTube, no activity) had a similar effect, our results indicate that the combination of short videos and rapid context-switching impairs intention recall and execution. We contribute a quantified understanding of the effect of social media feed format on Prospective Memory and outline consequences for media technology designers to not harm the users’ memory and wellbeing.

Human-in-the-loop optimization utilizes human expertise to guide machine optimizers iteratively and search for an optimal solution in a solution space. While prior empirical studies mainly investigated novices, we analyzed the impact of the levels of expertise on the outcome quality and corresponding subjective satisfaction. We conducted a study (N=60) in text, photo, and 3D mesh optimization contexts. We found that novices can achieve an expert level of quality performance, but participants with higher expertise led to more optimization iteration with more explicit preference while keeping satisfaction low. In contrast, novices were more easily satisfied and terminated faster. Therefore, we identified that experts seek more diverse outcomes while the machine reaches optimal results, and the observed behavior can be used as a performance indicator for human-in-the-loop system designers to improve underlying models. We inform future research to be cautious about the impact of user expertise when designing human-in-the-loop systems.

Dynamic balance is an essential skill for the human upright gait; therefore, regular balance training can improve postural control and reduce the risk of injury. Even slight variations in walking conditions like height or ground conditions can significantly impact walking performance. Virtual reality is used as a helpful tool to simulate such challenging situations. However, there is no agreement on design strategies for balance training in virtual reality under stressful environmental conditions such as height exposure. We investigate how two different training strategies, imitation learning, and gamified learning, can help dynamic balance control performance across different stress conditions. Moreover, we evaluate the stress response as indexed by peripheral physiological measures of stress, perceived workload, and user experience. Both approaches were tested against a baseline of no instructions and against each other. Thereby, we show that a learning-by-imitation approach immediately helps dynamic balance control, decreases stress, improves attention focus, and diminishes perceived workload. A gamified approach can lead to users being overwhelmed by the additional task. Finally, we discuss how our approaches could be adapted for balance training and applied to injury rehabilitation and prevention.