This paper presents the first systematic investigation of how congruence in visual self-representation influences self-perception and behavior. We span a continuum from the physical self through avatars with graded self-similarity to clearly dissimilar avatars in virtual reality (VR). In a 1 ×4 within-user study, participants completed movement and quiz tasks in either physical reality or a digital twin environment in VR, where they embodied one of three avatars: a photorealistic self-similar avatar, a dissimilar same-gender avatar, or a dissimilar opposite-gender avatar. Subjective measures included presence, sense of embodiment, self-identification, and perceived change, and were complemented by an objective movement metric of behavioral change. Compared to physical reality, VR, even with a self-similar avatar, produced lower presence, a weaker sense of embodiment, and reduced self-identification, revealing a persistent gap in visual congruence. Within VR, self-similar avatars enhanced body ownership, self-location, and self-identification relative to dissimilar avatars. Conversely, dissimilar avatars produced measurable behavioral changes compared with self-similar ones. Gender cues, however, had little impact in gender-neutral tasks. Overall, the findings show that photorealistic self-similar avatars reinforce embodiment and self-identification. However, VR still falls short of achieving congruence with physical reality, underscoring key challenges for avatar realism and ecological validity.

Integrated Virtual Reality (IVR) systems are central to avatar-mediated use cases in Virtual Reality (VR), reconstructing users' movements on avatars. They differ primarily in their tracking architectures, which determine how completely and accurately users' movements are captured and reconstructed on avatars. Many current IVR systems reduce user-worn hardware, trading reconstruction accuracy against cost and setup complexity, yet their impact on user experience and task performance across use cases remains underexplored. We compared three reduced user-worn IVR systems. Each system has distinct technical approaches: (1) Captury (markerless outside-in optical tracking), (2) Meta Movement SDK (markerless inside-out optical tracking), and (3) Vive Trackers (marker-based outside-in optical tracking with IMUs). In a 3x5 mixed-design, participants performed five tasks, simulating different use cases, to probe distinct aspects of these systems. No system consistently outperformed the others. Meta excelled in hand-based, fast-paced interactions, while Captury and Vive performed better in lower-body tasks and during full-body pose observation. These findings underscore the need to evaluate reduced user-worn IVR systems within the specific use case. We offer practical guidance for system selection based on use-case demands and released our tasks as an open-source, extensible framework to support future evaluations for selecting IVR systems.

Social Virtual Reality (SVR) allows users to meet and build relationships through embodied avatars and real-time interaction in virtual spaces. While embodiment can strengthen social connections and presence, it can also intensify negative encounters, making SVR particularly vulnerable to harassment. Despite frequent reports of verbal, visual, and "physical" violations in SVR, little is known about how harassment reshapes users' self-perception, including their sense of embodiment, self-identification, closeness, and avatar customization preferences. We conducted a controlled experiment with 52 participants who experienced either a neutral or a harassment condition in a scenario modeled after real SVR incidents. Participants perceived the harassing peer as significantly more negative, annoying, and disturbing than the neutral peer. Contrary to prior reports, harassment did not significantly affect well-being measures, including emotional state, self-esteem, and physiological arousal, within this controlled scenario. However, participants reported stronger bodily change, attributed more of their own attitudes and emotions to their avatars, and increased interpersonal distance when personal space was invaded. Self-reported coping strategies included ignoring, stepping back, using humor, and retaliating. Notably, avatar customization preferences shifted across conditions. Participants in the neutral condition favored personalized avatars, whereas those in the harassment condition more frequently preferred anonymity in public spaces. Together, these findings demonstrate that harassment in SVR not only exploits embodiment but also reshapes self-perception. We further contribute methodological insights into how harassment can be ethically and reproducibly studied in controlled SVR-like experiments.

Media bias is a multifaceted problem, leading to one-sided views and impacting decision-making. A way to address digital media bias is to detect and indicate it automatically through machine-learning methods. However, such detection is limited due to the difficulty of obtaining reliable training data. Human-in-the-loop-based feedback mechanisms have proven an effective way to facilitate the data-gathering process. Therefore, we introduce and test feedback mechanisms for the media bias domain, which we then implement on NewsUnfold, a news-reading web application to collect reader feedback on machine-generated bias highlights within online news articles. Our approach augments dataset quality by significantly increasing inter-annotator agreement by 26.31% and improving classifier performance by 2.49%. As the first human-in-the-loop application for media bias, the feedback mechanism shows that a user-centric approach to media bias data collection can return reliable data while being scalable and evaluated as easy to use. NewsUnfold demonstrates that feedback mechanisms are a promising strategy to reduce data collection expenses and continuously update datasets to changes in context.

Virtual reality (VR) exercise applications are promising tools, e.g., for at-home training and rehabilitation. However, existing applications vary significantly in key design choices such as environments, embodiment, and virtual coaching, making it difficult to derive clear design guidelines. A prominent design choice is the use of embodied virtual coaches, which guide user interaction and provide feedback. In a user study with 76 participants, we investigated how different levels of performance specificity in feedback from an embodied virtual coach affect intermediate factors, such as VR experience, motivation, and coach perception. Participants performed lower-body movement exercises, i.e., Leg Raises and Knee Extensions, commonly used in knee rehabilitation. We found that highly performance-specific feedback led to higher scores compared to medium specificity for perceived realism, as well as the anthropomorphism and sympathy of the virtual coach, but did not affect motivation. Based on our findings, we propose the design suggestion to include precise, performance-specific details when creating feedback for a virtual coach. We observed a descriptive pattern of higher scores in the low specificity condition compared to the medium condition on most measures, which raises the possibility that less specific feedback may, in some cases, be perceived more positively than moderately specific feedback. These findings provide valuable insights into how design choices impact relevant intermediate factors that are crucial for maximizing both workout effectiveness and the quality of the virtual coaching experience.

Virtual Reality Exposure Therapy (VRET) has become an effective, customizable, and affordable treatment for various psychological and physiological disorders. Specifically, it is used to treat specific anxiety disorders, such as acrophobia or arachnophobia, for decades. However, to ensure a positive outcome for patients, we must understand and control the effects potentially caused by the technology and medium of Virtual Reality (VR) itself. This article specifically investigates the impact of the Plausibility illusion (Psi), as one of the two theorized presence components, on the fear of heights. In two experiments, 30 participants each experienced two different heights with congruent and incongruent object behaviors in a 2 x 2 within-subject design. Results show that the strength of the congruence manipulation plays a significant role. Only when incongruencies are strong enough will they be recognized by users, specifically in high fear conditions, as triggered by exposure to increased heights. If incongruencies are too subtle, they seem to be overshadowed by the stronger fear reactions. Our evidence contributes to recent theories of VR effects and emphasizes the importance of understanding and controlling factors potentially assumed to be incidental, specifically during VRET designs. Incongruencies should be controlled so that they do not have an unwanted influence on the patient’s fear response.

Interpupillary distance (IPD) is the most important parameter for creating a user-specific stereo parallax, which in turn is crucial for correct depth perception. This is why contemporary Head-Mounted Displays (HMDs) offer adjustable lenses to adapt to users’ individual IPDs. However, today’s Video See-Through Augmented Reality (VST AR) HMDs use fixed camera placements to reconstruct the stereoscopic view of a user’s environment. This leads to a potential mismatch between individual IPD settings and the fixed Inter-Camera Distances (ICD), which can lead to perceptual incongruencies, limiting the usability and, potentially, the applicability of VST AR in depth-sensitive use cases. To investigate this incongruency between IPD and ICD, we conducted a 2 × 3 mixed-factor design user study using a near-field, open-loop reaching task comparing distance judgments of Virtual Reality (VR) and VST AR. We also investigated changes in reaching performance via perceptual calibration by incorporating a feedback phase between pre- and post-phase conditions, with a particular focus on the influence of IPD-ICD differences. Our Linear Mixed Model (LMM) analysis showed a significant difference between VR and VST AR, an effect of IPD-ICD mismatch, and a combined effect of both factors. However, subjective measures showed no effect underlining the subconscious nature of the perception of VST AR. This novel insight and its consequences are discussed specifically for depth perception tasks in AR, eXtended Reality (XR), and potential use cases.

Social virtual reality is an emerging medium of communication. In this medium, a user’s avatar (virtual representation) is controlled by the tracked motion of the user’s headset and hand controllers. This tracked motion is a rich data stream that can leak characteristics of the user or can be effectively matched to previously-identified data to identify a user. To better understand the boundaries of motion data identifiability, we investigate how varying training data duration and train-test delay affects the accuracy at which a machine learning model can correctly classify user motion in a supervised learning task simulating re-identification. The dataset we use has a unique combination of a large number of participants, long duration per session, large number of sessions, and a long time span over which sessions were conducted. We find that training data duration and train-test delay affect identifiability; that minimal train-test delay leads to very high accuracy; and that train-test delay should be controlled in future experiments.

The use of virtual and augmented reality devices is increasing, but these sensor-rich devices pose risks to privacy. The ability to track a user’s motion and infer the identity or characteristics of the user poses a privacy risk that has received significant attention. Existing deep-network-based defenses against this risk, however, require significant amounts of training data and have not yet been shown to generalize beyond specific applications. In this work, we study the effect of signal degradation on identifiability, specifically through added noise, reduced framerate, reduced precision, and reduced dimensionality of the data. Our experiment shows that state-of-the-art identification attacks still achieve near-perfect accuracy for each of these degradations. This negative result demonstrates the difficulty of anonymizing this motion data and gives some justification to the existing data- and compute-intensive deep-network based methods.

This paper investigates if and how self-similarity and having motor control impact sense of embodiment, self-identification, and body weight perception in Augmented Reality (AR). We conducted a 2x2 mixed design experiment involving 60 participants who interacted with either synchronously moving virtual humans or independently moving ones, each with self-similar or generic appearances, across two consecutive AR sessions. Participants evaluated their sense of embodiment, self-identification, and body weight perception of the virtual human. Our results show that self-similarity significantly enhanced sense of embodiment, self-identification, and the accuracy of body weight estimates with the virtual human. However, the effects of having motor control over the virtual human movements were notably weaker in these measures than in similar VR studies. Further analysis indicated that not only the virtual human itself but also the participants' body weight, self-esteem, and body shape concerns predict body weight estimates across all conditions. Our work advances the understanding of virtual human body weight perception in AR systems, emphasizing the importance of factors such as coherence with the real-world environment.

Facial expressions are crucial for many eXtended Reality (XR) use cases, from mirrored self exposures to social XR, where users interact via their avatars as digital alter egos. However, current XR devices differ in sensor coverage of the face region. Hence, a faithful reconstruction of facial expressions either has to exclude these areas or synthesize missing animation data with model-based approaches, potentially leading to perceivable mismatches between executed and perceived expression. This paper investigates potential effects of the coverage of facial animations (none, partial, or whole) on important factors of self-perception. We exposed 83 participants to their mirrored personalized avatar. They were shown their mirrored avatar face with upper and lower face animation, upper face animation only, lower face animation only, or no face animation. Whole animations were rated higher in virtual embodiment and slightly lower in uncanniness. Missing animations did not differ from partial ones in terms of virtual embodiment. Contrasts showed significantly lower humanness, lower eeriness, and lower attractiveness for the partial conditions. For questions related to self-identification, effects were mixed. We discuss participants' shift in body part attention across conditions. Qualitative results show participants perceived their virtual representation as fascinating yet uncanny.

This study explores how people interact in dyadic social eXtended Reality (XR), focusing on two main factors: the animation type of a conversation partner's avatar and how immersed the user feels in the virtual environment. Specifically, we investigate how 1) idle behavior, 2) AI-generated gestures, and 3) motion-captured movements from a confederate (a controlled partner in the study) influence the quality of conversation and how that partner is perceived. We examined these effects in both symmetric interactions (where both participants use VR headsets and controllers) and asymmetric interactions (where one participant uses a desktop setup). We developed a social XR platform that supports asymmetric device configurations to provide varying levels of immersion. The platform also supports a modular avatar animation system providing idle behavior, real-time AI-generated co-speech gestures, and full-body motion capture. Using a 2×3 mixed design with 39 participants, we measured users' sense of spatial presence, their perception of the confederate, and the overall conversation quality. Our results show that users who were more immersed felt a stronger sense of presence and viewed their partner as more human-like and believable. Surprisingly, however, the type of avatar animation did not significantly affect conversation quality or how the partner was perceived. Participants often reported focusing more on what was said rather than how the avatar moved.

Immersive 3D mid-air sketching systems liberate users from the confines of traditional 2D sketching canvases. However, complications from perceptual challenges in Virtual Reality (VR), combined with the ergonomic and cognitive challenges of sketching in all three dimensions in mid-air lower the accuracy and aesthetic quality of 3D sketches. This paper explores how color-to-depth and opacity cues support users to create and perceive freehand, 3D strokes in room-scale sketching, unlocking a full 360° of freedom for creation. We implemented three graphic depth shader cues modifying the (1) alpha, (2) hue, and (3) value levels of a single color to dynamically adjust the color and transparency of meshes relative to their depth from the user. We investigated how these depth cues influence sketch efficiency, sketch quality, and total sketch experience with 24 participants in a comparative, counterbalanced, 4 x 1 within-subjects user study. First, with our graphic depth shader cues we could successfully transfer results of prior research in seated sketching tasks to room-scale scenarios. Our color-to-depth cues improved the similarity of sketches to target models. This highlights the usefulness of the color-to-depth approach even for the increased range of motion and depth in room-scale sketching. Second, our shaders assisted participants to complete tasks faster, spend a greater percentage of task time sketching, reduced the feeling of mental tiredness and improved the feeling of sketch efficiency in room-scale sketching. We discuss these findings and share our insights and conclusions to advance the research on improving spatial cognition in immersive sketching systems.

In recent years, the demand for interactive photorealistic three-dimensional (3D) environments has increased in various fields, including architecture, engineering, and entertainment. However, achieving a balance between the quality and efficiency of high-performance 3D applications and virtual reality (VR) remains challenging. This study addresses this issue by revisiting and extending view interpolation for image-based rendering (IBR), which enables the exploration of spacious open environments in 3D and VR. Therefore, we introduce multimorphing, a novel rendering method based on the spatial data structure of 2D image patches, called the image graph. Using this approach, novel views can be rendered with up to six degrees of freedom using only a sparse set of views. The rendering process does not require 3D reconstruction of the geometry or per-pixel depth information, and all relevant data for the output are extracted from the local morphing cells of the image graph. The detection of parallax image regions during preprocessing reduces rendering artifacts by extrapolating image patches from adjacent cells in real-time. In addition, a GPU-based solution was presented to resolve exposure inconsistencies within a dataset, enabling seamless transitions of brightness when moving between areas with varying light intensities. Experiments on multiple real-world and synthetic scenes demonstrate that the presented method achieves high "VR-compatible" frame rates, even on mid-range and legacy hardware, respectively. While achieving adequate visual quality even for sparse datasets, it outperforms other IBR and current neural rendering approaches. Using the correspondence-based decomposition of input images into morphing cells of 2D image patches, multidimensional image morphing provides high-performance novel view generation, supporting open 3D and VR environments. Nevertheless, the handling of morphing artifacts in the parallax image regions remains a topic for future research.

Virtual reality (VR) offers various opportunities for innovative therapeutic approaches, especially regarding self-related mind-body interventions. We introduce a VR body swap system enabling multiple users to swap their perspectives and appearances and evaluate its effects on virtual sense of embodiment (SoE) and perception- and cognition-based self-related processes. In a self-compassion-framed scenario, twenty participants embodied their personalized, photorealistic avatar, swapped bodies with an unfamiliar peer, and reported their SoE, interoceptive awareness (perception), and self-compassion (cognition). Participants’ experiences differed between bottom-up and top-down processes. Regarding SoE, their agency and self-location shifted to the swap avatar, while their top-down self-identification remained with their personalized avatar. Further, the experience positively affected interoceptive awareness but not self-compassion. Our outcomes offer novel insights into the SoE in a multiple-embodiment scenario and highlight the need to differentiate between the different processes in intervention design. They raise concerns and requirements for future research on avatar-based mind-body interventions.

This work presents a study where we used incongruencies on the cognitive and the perceptual layer to investigate their effects on perceived plausibility and, thereby, presence and spatial presence. We used a $2 \times 3$ within-subject design with the factors familiar size (cognitive manipulation) and immersion (perceptual manipulation). For the different levels of immersion, we implemented three different tracking qualities: rotation-and-translation tracking, rotation-only tracking, and stereoscopic-view-only tracking. Participants scanned products in a virtual supermarket where the familiar size of these objects was manipulated. Simultaneously, they could either move their head normally or need to use the thumbsticks to navigate their view of the environment. Results show that both manipulations had a negative effect on perceived plausibility and, thereby, presence. In addition, the tracking manipulation also had a negative effect on spatial presence. These results are especially interesting in light of the ongoing discussion about the role of plausibility and congruence in evaluating XR environments. The results can hardly be explained by traditional presence models, where immersion should not be an influencing factor for perceived plausibility. However, they are in agreement with the recently introduced Congruence and Plausibility (CaP) model and provide empirical evidence for the model’s predicted pathways.

The sense of presence and the sense of embodiment are two fundamental qualia, pivotal to many virtual reality experiences. Empirical research indicates a notable interdependence between these two qualia, where manipulations designed to affect one often exhibit a concurrent influence on the other. Existing theories on the development of qualia in virtual reality make no or only insufficient statements on this deep interdependence. In this work, we present a novel theoretical perspective on this connection. Based on existing theories, we argue that all the fundamental cues influencing one quale have the potential to impact the other one too. We present three studies ($n=42, n=42, n=32$) that generally support this novel perspective. Among other things, they show that traditional spatial presence cues such as head-tracking and passive depth cues (stereoscopy, linear perspective, etc.) can potentially also serve as embodiment cues. Conversely, they show that typical embodiment cues such as the visuotactile and visuoproprioceptive synchrony of a virtual hand are also spatial presence cues. The cues only differ in terms of how strongly they influence the respective quale. This novel perspective not only enhances our understanding of fundamental mechanics of virtual reality but it can also guide the development of more effective measurement instruments.

In social virtual reality (social VR), users are threatened by potential cybercrimes, such as identity theft, sensitive data breaches, and embodied harassment. These concerns are heightened by the increasing interest in the metaverse, the advancements in photorealistic 3D user reconstructions, and the rising incidents of online privacy violations. Designing secure social VR applications that protect users while enhancing their experience, acceptance and trust remains a challenge. This article investigates potential identity management solutions in social VR, and their impacts on usability and user acceptance. We developed a social VR prototype with novel and established countermeasures, including motion biometric verification, and conducted a study with 52 participants. Our findings reveal diverse preferences for identity management and underscore the importance of authenticity, autonomy, and reciprocity. Key findings include: passive verification is favored for pragmatic user experience, while active verification is preferred for its hedonic quality; continuous or periodic verification strengthens users’ confidence in their privacy; and while user awareness promotes authentic engagement, it may also diminish the willingness to disclose personal information. This research not only offers foundational insights into the evaluated scenarios and countermeasures, but also sheds light on the designs of more trustworthy and inclusive social VR applications.

This work evaluates how the asymmetry of device configurations and verbal communication influence the user experience of social eXtended Reality (XR) for self-perception, other-perception, and task perception. We developed an application that enables social collaboration between two users with varying device configurations.We compare the conditions of one symmetric interaction, where both device configurations are Head-Mounted Displays (HMDs) with tracked controllers, with the conditions of one asymmetric interaction, where one device configuration is an HMD with tracked controllers and the other device configuration is a desktop screen with a mouse. In our study, 52 participants collaborated in a dyadic interaction on a sorting task while talking to each other. We compare our results to previous work that evaluated the same scenario without verbal communication. In line with prior research, self-perception is influenced by the immersion of the used device configuration and verbal communication. While co-presence was not affected by the device configuration or the inclusion of verbal communication, social presence was only higher for HMD configurations that allowed verbal communication. Task perception was hardly affected by the device configuration or verbal communication. We conclude that the device in social XR is important for self-perception with or without verbal communication. However, the results indicate that the device configuration only affects the qualities of social interaction in collaborative scenarios when verbal communication is enabled. To sum up, asymmetric collaboration maintains the high quality of self-perception and interaction for highly immersed users while still enabling the participation of less immersed users.

Immersive 3D sketching systems empower users with tools to create sketches directly in the air around themselves, in all three dimensions, using only simple hand gestures. These sketching systems have the potential to greatly extend the interactive capabilities of immersive learning environments. The perceptual challenges of Virtual Reality (VR), however, combined with the ergonomic and cognitive challenges of creating mid-air 3D sketches reduce the effectiveness of immersive sketching used for problem-solving, reflection, and to capture fleeting ideas. We contribute to the understanding of the potential challenges of mid-air sketching systems in educational settings, where expression is valued higher than accuracy, and sketches are used to support problem-solving and to explain abstract concepts. We conducted an empirical study with 36 participants with different spatial abilities to investigate if the way that people sketch in mid-air is dependent on the goal of the sketch. We compare the technique, quality, efficiency, and experience of participants as they create 3D mid-air sketches in three different tasks. We examine how users approach mid-air sketching when the sketches they create serve to convey meaning and when sketches are merely reproductions of geometric models created by someone else. We found that in tasks aimed at expressing personal design ideas, between starting and ending strokes, participants moved their heads more and their controllers at higher velocities and created strokes in faster times than in tasks aimed at recreating 3D geometric figures. They reported feeling less time pressure to complete sketches but redacted a larger percentage of strokes. These findings serve to inform the design of creative virtual environments that support reasoning and reflection through mid-air sketching. With this work, we aim to strengthen the power of immersive systems that support midair 3D sketching by exploiting natural user behavior to assist users to more quickly and faithfully convey their meaning in sketches.

Virtual environments (VEs) can be designed to evoke specific emotions for example by using colored light, not only applicable for games but also for virtual storytelling with a single storyteller. Social robots are perfectly suited as storytellers due to their multimodality. However, there is no research yet on the transferability of robotic storytelling to virtual reality (VR). In addition, the transfer of concepts from VE design such as adaptive room illumination to robotic storytelling has yet not been tested. Thus, we conducted a study comparing the same robotic storytelling with a physically embodied robotic storyteller and in VR to investigate the transferability of robotic storytelling to VR. As a second factor, we manipulated the room light following design guidelines for VEs or kept it constant. Results show that a virtual robotic storyteller is not perceived worse than a physically embodied storyteller, suggesting the applicability of virtual static robotic storytellers. Regarding emotion-driven lighting, no significant effect of colored lights on self-reported emotions was found, but adding colored light increased the social presence of the robot and its’ perceived competence in both VR and reality. As our study was limited by a static robotic storyteller not using bodily expressiveness future work is needed to investigate the interaction between well-researched robot modalities and the rather new modality of colored light based on our results.

This article investigates the influence of input/output device characteristics and degrees of immersion on the User Experience (UX) of specific eXtended Reality (XR) effects, i.e., presence, self-perception, other-perception, and task perception. It targets universal access to social XR, where dedicated XR hardware is unavailable or can not be used, but participation is desirable or even necessary. We compare three different device configurations: (i) desktop screen with mouse, (ii) desktop screen with tracked controllers, and (iii) Head-Mounted Display (HMD) with tracked controllers. 87 participants took part in collaborative dyadic interaction (a sorting task) with asymmetric device configurations in a specifically developed social XR. In line with prior research, the sense of presence and embodiment were significantly lower for the desktop setups. However, we only found minor differences in task load and no differences in usability and enjoyment of the task between the conditions. Additionally, the perceived humanness and virtual human plausibility of the other were not affected, no matter the device used. Finally, there was no impact regarding co-presence and social presence independent of the level of immersion of oneself or the other. We conclude that the device in social XR is important for self-perception and presence. However, our results indicate that the devices do not affect important UX and usability aspects, specifically, the qualities of social interaction in collaborative scenarios, paving the way for universal access to social XR encounters and significantly promoting participation.

Depending on their formality, clothes do not only change one’s appearance, but can also influence behavior and cognitive processes. Shoes are a special aspect of an outfit. Besides coming in various degrees of formality, their structure can affect human gait. Avatars used to embody users in immersive Virtual Reality (VR) can wear any kind of clothing. According to the Proteus Effect, the appearance of a user’s avatar can influence their behavior. Users change their behavior in accordance to the expected behavior of the avatar. In our study, we embody 39 participants with a generic avatar of the user’s gender wearing three different pairs of shoes as within condition. The shoes differ in degree of formality. We measure the gait during a 2-minute walking task during which participants wore the same real shoe and assess selective attention using the Stroop task. Our results show significant differences in gait between the tested virtual shoe pairs. We found small effects between the three shoe conditions with respect to selective attention. However, we found no significant differences with respect to correct items and response time in the Stroop task. Thus, our results indicate that virtual shoes are accepted by users and, although not eliciting any physical constraints, lead to changes in gait. This suggests that users not only adjust personal behavior according to the Proteus Effect, but also are affected by virtual biomechanical constraints. Also, our results suggest a potential influence of virtual clothing on cognitive performance.

The Field of View (FoV) is a central technical display characteristic of Head-Mounted Displays (HMDs), which has been shown to have a notable impact on important aspects of the user experience. For example, an increased FoV has been shown to foster a sense of presence and improve peripheral information processing, but it also increases the risk of VR sickness. This article investigates the impact of a wider but inhomogeneous FoV on the perceived plausibility, measuring its effects on presence, spatial presence, and VR sickness as a comparison to and replication of effects from prior work. We developed a low-resolution peripheral display extension to pragmatically increase the FoV, taking into account the lower peripheral acuity of the human eye. While this design results in inhomogeneous resolutions of HMDs at the display edges, it also is a low complexity and low-cost extension. However, its effects on important VR qualities have to be identified. We conducted two experiments with 30 and 27 participants, respectively. In a randomized 2x3 within-subject design, participants played three rounds of bowling in VR, both with and without the display extension. Two rounds contained incongruencies to induce breaks in plausibility. In experiment 2, we enhanced one incongruency to make it more noticeable and improved the shortcomings of the display extension that had previously been identified. However, neither study measured the low-resolution FoV extension’s effect in terms of perceived plausibility, presence, spatial presence, or VR sickness. We found that one of the incongruencies could cause a break in plausibility without the extension, confirming the results of a previous study.

Wave-optical phenomena, such as diffraction, significantly impact the visual appearance of surfaces. Despite their importance, wave-optical reflection models are rare and computationally expensive. Recently, we presented a real-time model that accounts for diffraction-induced color shifts and speckle. Given that diffraction phenomena are highly dependent on illumination and viewing directions, as well as stereoscopic vision, we developed a VR demo to evaluate the new model. This demo shows the substantial impact of diffraction on the appearance of rough surfaces, particularly in stereoscopic viewing.

This paper presents an openly available implementation of a modular pipeline architecture for character animation. It effectively decomposes frequently necessary processing steps into dedicated character processors, such as copying data from various motion sources, applying inverse kinematics, or scaling the character. Processors can easily be parameterized, extended (e.g., with AI), and freely arranged or even duplicated in any order necessary, greatly reducing side effects and fostering fine-tuning, maintenance, and reusability of the complex interplay of real-time animation steps.

This paper introduces “Motion Passwords”, a novel biometric authentication approach where virtual reality users verify their identity by physically writing a chosen word in the air with their hand controller. This method allows combining three layers of verification: knowledge-based password input, handwriting style analysis, and motion profile recognition. As a first step towards realizing this potential, we focus on verifying users based on their motion profiles. We conducted a data collection study with 48 participants, who performed over 3800 Motion Password signatures across two sessions. We assessed the effectiveness of feature-distance and similarity-learning methods for motion-based verification using the Motion Passwords as well as specific and uniform ball-throwing signatures used in previous works. In our results, the similarity-learning model was able to verify users with the same accuracy for both signature types. This demonstrates that Motion Passwords, even when applying only the motion-based verification layer, achieve reliability comparable to previous methods. This highlights the potential for Motion Passwords to become even more reliable with the addition of knowledge-based and handwriting style verification layers. Furthermore, we present a proof-of-concept Unity application demonstrating the registration and verification process with our pretrained similarity-learning model. We publish our code, the Motion Password dataset, the pretrained model, and our Unity prototype on https://github.com/cschell/MoPs

This paper presents a novel experiment investigating the relationship between virtual disembodiment and time perception in Virtual Reality (VR). Recent work demonstrated that the absence of a virtual body in a VR application changes the perception of time. However, the effects of simulating an out-of-body experience (OBE) in VR on time perception are still unclear. We designed an experiment with two types of virtual disembodiment techniques based on viewpoint gradual transition: a virtual body’s behind view and facing view transitions. We investigated their effects on forty-four participants in an interactive scenario where a lamp was repeatedly activated and time intervals were estimated. Our results show that, while both techniques elicited a significant virtual disembodiment perception, time duration estimations in the minute range were only shorter in the facing view compared to the eye view condition. We believe that reducing agency in the facing view is a key factor in the time perception alteration. This provides first steps towards a novel approach to manipulating time perception in VR, with potential applications for mental health treatments such as schizophrenia or depression and for improving our understanding of the relation between body, virtual body, and time.

Recent research shows that visualizing linguistic bias mitigates its negative effects. However, reliable automatic detection methods to generate such visualizations require costly, knowledge-intensive training data. To facilitate data collection for media bias datasets, we present News Ninja, a game employing data-collecting game mechanics to generate a crowdsourced dataset. Before annotating sentences, players are educated on media bias via a tutorial. Our findings show that datasets gathered with crowdsourced workers trained on News Ninja can reach significantly higher inter-annotator agreements than expert and crowdsourced datasets with similar data quality. As News Ninja encourages continuous play, it allows datasets to adapt to the reception and contextualization of news over time, presenting a promising strategy to reduce data collection expenses, educate players, and promote long-term bias mitigation.

Our work investigates the influence of self-related cues in the design of virtual humans on body perception in virtual reality. In a $2\times 2$ mixed design, 64 participants faced photorealistic virtual humans either as a motion-synchronized embodied avatar or as an autonomous moving agent, appearing subsequently with a personalized and generic texture. Our results unveil that self-related cues through embodiment and personalization yield an individual and complemented increase in participants' sense of embodiment and self-identification towards the virtual human. Different body weight modification and estimation tasks further showed an impact of both factors on participants' body weight perception. Additional analyses revealed that the participant's body mass index predicted body weight estimations in all conditions and that participants' self-esteem and body shape concerns correlated with different body weight perception results. Hence, we have demonstrated the occurrence of double standards through induced self-related cues in virtual human perception, especially through embodiment.

Text input is desirable across various eXtended Reality (XR) use cases and is particularly crucial for knowledge and office work. This article compares handwriting text input between Virtual Reality (VR) and Video See-Through Augmented Reality (VST AR), facilitated by physically aligned and mid-air surfaces when writing simple and complex sentences. In a $2\times 2\times 2$ experimental design, 72 participants performed two ten-minute handwriting sessions, each including ten simple and ten complex sentences representing text input in real-world scenarios. Our developed handwriting application supports different XR displays, surface alignments, and handwriting recognition based on digital ink. We evaluated usability, user experience, task load, text input performance, and handwriting style. Our results indicate high usability with a successful transfer of handwriting skills to the virtual domain. XR displays and surface alignments did not impact text input speed and error rate. However, sentence complexities did, with participants achieving higher input speeds and fewer errors for simple sentences (17.85 WPM, 0.51% MSD ER) than complex sentences (15.07 WPM, 1.74% MSD ER). Handwriting on physically aligned surfaces showed higher learnability and lower physical demand, making them more suitable for prolonged handwriting sessions. Handwriting on mid-air surfaces yielded higher novelty and stimulation ratings, which might diminish with more experience. Surface alignments and sentence complexities significantly affected handwriting style, leading to enlarged and more connected cursive writing in both mid-air and for simple sentences. The study also demonstrated the benefits of using XR controllers in a pen-like posture to mimic styluses and pressure-sensitive tips on physical surfaces for input detection. We additionally provide a phrase set of simple and complex sentences as a basis for future text input studies, which can be expanded and adapted.

Spatial User Interfaces along the Reality-Virtuality continuum heavily depend on accurate depth perception. However, current display technologies still exhibit shortcomings in the simulation of accurate depth cues, and these shortcomings also vary between Virtual or Augmented Reality (VR, AR: eXtended Reality (XR) for short). This article compares depth perception between VR and Video See-Through (VST) AR. We developed a digital twin of an existing office room where users had top erform five depth-dependent tasks in VR and VST AR. Thirty-two participants took part in a user study using a 1 × 4 within-subjects design. Our results reveal higher misjudgment rates in VST AR due to conflicting depth cues between virtual and physical content. Increased head movements observed in participants were interpreted as a compensatory response to these conflicting cues. Furthermore, a longer task completion time in the VST AR condition indicates a lower task performance in VST AR. Interestingly, while participants rated the VR condition as easier and contrary to the increased misjudgments and lower performance with the VST AR display, a majority still expressed a preference for the VST AR experience. We discuss and explain these findings with the high visual dominance and referential power of the physical content in the VST AR condition, leading to a higher spatial presence and plausibility.

Virtual Reality (VR) systems rely on established real-time rendering techniques to uphold a strict performance budget of a few milliseconds per frame. The utilized techniques are based on approximations that trade speed-up for the correctness of the generated depth cues. The visual impact of these trade-offs, however, remains largely unexplored in the context of VR. In this paper we focus specifically on the perception of distorted specular reflections. Our research goal is to quantify the impairment of common reflection approximations on the resulting visual quality in VR. To this end, a subjective quality assessment is conducted, where participants rate the visual difference and visual annoyance of multiple established reflection approximations in a side-by-side comparison to a raytraced reference solution. We introduce the first dataset of its kind, containing 9 specular reflection approximations evaluated for two object types and three material smoothness values. Our results show the quality benefits associated with currently used reflection methods, and we discuss the implications for rendering VR environments.

Virtual Reality (VR) allows us to replace our visible body with a virtual self-representation (avatar) and to explore its effects on our body perception. While the feeling of owning and controlling a virtual body is widely researched, how VR affects the awareness of internal body signals (body awareness) remains open. Forty participants performed moving meditation tasks in reality and VR, either facing their mirror image or not. Both the virtual environment and avatars photorealistically matched their real counterparts. We found a negative effect of VR on body awareness, mediated by feeling embodied in and changed by the avatar. Further, we revealed a negative effect of a mirror on body awareness. Our results indicate that assessing body awareness should be essential in evaluating VR designs and avatar embodiment aiming at mental health, as even a scenario as close to reality as possible can distract users from their internal body signals.

Currently, there is an ongoing debate about the influencing factors of one’s extended reality (XR) experience. Plausibility, congruence, and their role have recently gained more and more attention. One of the latest models to describe XR experiences, the Congruence and Plausibility model (CaP), puts plausibility and congruence right in the center. However, it is unclear what influence they have on the overall XR experience and what influences our perceived plausibility rating. In this paper, we implemented four different incongruencies within a virtual reality scene using breaks in plausibility as an analogy to breaks in presence. These manipulations were either located on the cognitive or perceptual layer of the CaP model. They were also either connected to the task at hand or not. We tested these manipulations in a virtual bowling environment to see which influence they had. Our results show that manipulations connected to the task caused a lower perceived plausibility. Additionally, cognitive manipulations seem to have a larger influence than perceptual manipulations. We were able to cause a break in plausibility with one of our incongruencies. These results show a first direction on how the influence of plausibility in XR can be systematically investigated in the future.

Body awareness is relevant for the efficacy of psychotherapy. However, previous work on virtual reality (VR) and avatar-assisted therapy has often overlooked it. We investigated the effect of avatar individualization on body awareness in the context of VR-specific user experience, including sense of embodiment (SoE), plausibility, and sense of presence (SoP). In a between-subject design, 86 participants embodied three avatar types and engaged in VR movement exercises. The avatars were (1) generic and gender-matched, (2) customized from a set of pre-existing options, or (3) personalized photorealistic scans. Compared to the other conditions, participants with personalized avatars reported increased SoE, yet higher eeriness and reduced body awareness. Further, SoE and SoP positively correlated with body awareness across conditions. Our results indicate that VR user experience and body awareness do not always dovetail and do not necessarily predict each other. Future research should work towards a balance between body awareness and SoE.

The utilization of virtual reality (VR) technology has shown promise in various therapeutic applications, particularly in exposure therapy for reducing fear of certain situations objects or activities, e.g. fear of height, or negative evaluation of others in social situations. VR has been shown to yield positive outcomes in follow-up studies, and provides a safe and ecological therapeutic environment for therapists and their patients. This paper presents a collaborative effort to develop a VR speech therapy system which simulates a virtual audience for users to practice their public speaking skills. We describe a novel web-based graphical user interface that enables therapists to manage the therapy session using a simple timeline. Lastly, we present the results from a qualitative study with therapists and teachers with functional dysphonia, which highlight the potential of such an application to support and augment the therapists' work and the remaining challenges regarding the design of natural interactions, agent behaviours and scenario customisation for patients.

Machine learning is an essential aspect of modern life that many educational institutions incorporate into their curricula. Often, students struggle to grasp how neural networks learn. Teaching these concepts could be assisted with pedagogical agents and serious games, which both have proven helpful for complex topics like engineering. We present "Traversing the Pass," a serious game that utilizes a mentor-like agent to explain the underlying machine learning concepts and provides feedback. We optimized the agent's design in a pre-study before evaluating its effectiveness compared to a text-only user interface with experts and students. Participants performed better in a second assessment two weeks later if they played the game using the agent. Although criticized as repetitive, the game created an understanding of basic machine learning concepts and achieved high flow values. Our results indicate that agents could be used to enhance the beneficial effects of serious games with improved knowledge retention.

This demo presents an innovative and validated method for measuring interpersonal trust in virtual humans as social interaction partners. In our design, participants are tasked to navigate through a virtual reality maze and interact with a virtual human, seeking advice on which path to take. By carefully designing the experiments and context, we induced participants to believe that they are interacting with a real person. A validation study, manipulating virtual humans' appearance, tone of voice, and engagement, revealed that the frequency of advice-seeking in the maze correlates with participants' evaluation of interpersonal trust towards virtual humans. Thus, our paradigm represents a valuable research tool for studying trust in virtual reality.

This paper presents the development and preliminary evaluation of a personality enhancer behaviour model for virtual audiences to increase their realism and individualism. We conducted a systematic literature review and identified sixteen posture parameters to modify seated animations dynamically, calling them personality enhancers. We grouped them into four main categories: body, gaze, face and gesture behaviour modifiers. We implemented a unique animation modifier system on top of a game engine to apply these personality enhancers on existing pre-recorded generic seated animations. The first results with sixty participants in an online video survey show that the model can successfully simulate individuals with low and high levels of extroversion as well as with low and high levels of emotional stability.

This paper investigates the relationship between perceived object motion and the experience of time in virtual environments. We developed an application to measure how the motion properties of virtual objects and the degree of immersion and embodiment may affect the time experience. A first study (n = 145) was conducted remotely using an online video survey, while a second study (n = 60) was conducted under laboratory conditions in virtual reality (VR). Participants in both studies experienced seven different virtual objects in a randomized order and then answered questions about time experience. The VR study added an "embodiment" condition in which participants were either represented by a virtual full body or lacked any form of virtual body representation. In both studies, time was judged to pass faster when viewing oscillating motion in immersive and non-immersive settings and independently of the presence or absence of a virtual body. This trend was strongest when virtual pendulums were displayed. Both studies also found a significant inverse correlation between the passage of time and boredom. Our results support the development of applications that manipulate the perception of time in virtual environments for therapeutic use, for instance, for disorders such as depression, autism, and schizophrenia. Disturbances in the perception of time are known to be associated with these disorders.

When blending virtual and physical content, certain incongruencies emerge from hardware limitations, inaccurate tracking, or different appearances of virtual and physical content. They restrain us from perceiving virtual and physical content as one experience. Hence, it is crucial to investigate these issues to determine how they influence our experience. We present a virtualized augmented reality simulation that can systematically examine single incongruencies or different configurations.

Abstract Medical graduates lack procedural skills experience required to manage emergencies. Recent advances in virtual reality (VR) technology enable the creation of highly immersive learning environments representing easy-to-use and affordable solutions for training with simulation. However, the feasibility in compulsory teaching, possible side effects of immersion, perceived stress, and didactic benefits have to be investigated systematically. VR-based training sessions using head-mounted displays alongside a real-time dynamic physiology system were held by student assistants for small groups followed by debriefing with a tutor. In the pilot study, 36 students rated simulation sickness. In the main study, 97 students completed a virtual scenario as active participants (AP) and 130 students as observers (OBS) from the first-person perspective on a monitor. Participants completed questionnaires for evaluation purposes and exploratory factor analysis was performed on the items. The extent of simulation sickness remained low to acceptable among participants of the pilot study. In the main study, students valued the realistic environment and guided practical exercise. AP perceived the degree of immersion as well as the estimated learning success to be greater than OBS and proved to be more motivated post training. With respect to AP, the factor “sense of control” revealed a typical inverse U-shaped relationship to the scales “didactic value” and “individual learning benefit”. Summing up, curricular implementation of highly immersive VR-based training of emergencies proved feasible and found a high degree of acceptance among medical students. This study also provides insights into how different conceptions of perceived stress distinctively moderate subjective learning success.

This article compares two state-of-the-art text input techniques between non-stationary virtual reality (VR) and video see-through augmented reality (VST AR) use-cases as XR display condition. The developed contact-based mid-air virtual tap and word-gesture (swipe) keyboard provide established support functions for text correction, word suggestions, capitalization, and punctuation. A user evaluation with 64 participants revealed that XR displays and input techniques strongly affect text entry performance, while subjective measures are only influenced by the input techniques. We found significantly higher usability and user experience ratings for tap keyboards compared to swipe keyboards in both VR and VST AR. Task load was also lower for tap keyboards. In terms of performance, both input techniques were significantly faster in VR than in VST AR. Further, the tap keyboard was significantly faster than the swipe keyboard in VR. Participants showed a significant learning effect with only ten sentences typed per condition. Our results are consistent with previous work in VR and optical see-through (OST) AR, but additionally provide novel insights into usability and performance of the selected text input techniques for VST AR. The significant differences in subjective and objective measures emphasize the importance of specific evaluations for each possible combination of input techniques and XR displays to provide reusable, reliable, and high-quality text input solutions. With our work, we form a foundation for future research and XR workspaces. Our reference implementation is publicly available to encourage replicability and reuse in future XR workspaces.

Virtual humans, including virtual agents and avatars, play an increasingly important role as VR technology advances. For example, virtual humans are used as digital bodies of users in social VR or as interfaces for AI assistants in online financing. Interpersonal trust is an essential prerequisite in real-life interactions, as well as in the virtual world. However, to date, there are no established interpersonal trust measurement tools specifically for virtual humans in virtual reality. This study fills the gap, by contributing a novel validated behavioural tool to measure interpersonal trust towards a specific virtual social interaction partner in social VR. This validated paradigm is inspired by a previously proposed virtual maze task that measures trust towards virtual characters. In the current study, a variant of this paradigm was implemented. The task of the users (the trustors) is to navigate through a maze in virtual reality, where they can interact with a virtual human (the trustee). They can choose to 1) ask for advice and 2) follow the advice from the virtual human if they want to. These measures served as behavioural measures of trust. We conducted a validation study with 70 participants in a between-subject design. The two conditions did not differ in the content of the advice but in the appearance, tone of voice and engagement of the trustees (alleged as avatars controlled by other participants). Results indicate that the experimental manipulation was successful, as participants rated the virtual human as more trustworthy in the trustworthy condition than in the untrustworthy condition. Importantly, this manipulation affected the trust behaviour of our participants, who, in the trustworthy condition, asked for advice more often and followed advice more often, indicating that the paradigm is sensitive to assessing interpersonal trust towards virtual humans. Thus, our paradigm can be used to measure differences in interpersonal trust towards virtual humans and may serve as a valuable research tool to study trust in virtual reality.

Photorealistic avatars show great potential in social VR and VR collaboration. However, identity and privacy issues are threatening avatars' authenticity in social VR. In addition to the necessary authentication and protection, effective solutions are needed to convey avatars' authenticity status to users and thereby enhance the overall trustworthiness. We designed several visual indicators (VIs) using static or dynamic visual effects on photorealistic avatars and evaluated their effectiveness in visualizing avatars' authenticity status. In this study we explored suitable attributes and designs for conveying the authenticity of photorealistic avatars and influencing their perceived trustworthiness. Furthermore, we investigated how different interactivity levels influence their effectiveness (the avatar was either presented in a static image, an animated video clip, or an immersive virtual environment). Our findings showed that using a full name can increase trust, while most other VIs could decrease users' trust. We also found that interactivity levels significantly impacted users' trust and the effectiveness of VIs. Based on our results, we developed design guidelines for visual indicators as effective tools to convey authenticity, as a first step towards the improvement of trustworthiness in social VR with identity management.

Many studies show the significance of the Proteus effect for serious virtual reality applications. The present study extends the existing knowledge by considering the relationship (congruence) between the self-embodiment (avatar) and the virtual environment. We investigated the impact of avatar and environment types and their congruence on avatar plausibility, sense of embodiment, spatial presence, and the Proteus effect. In a $2\times 2$ between-subjects design, participants embodied either an avatar in sports- or business wear in a semantic congruent or incongruent environment while performing lightweight exercises in virtual reality. The avatar-environment congruence significantly affected the avatar's plausibility but not the sense of embodiment or spatial presence. However, a significant Proteus effect emerged only for participants who reported a high feeling of (virtual) body ownership, indicating that a strong sense of having and owning a virtual body is key to facilitating the Proteus effect. We discuss the results assuming current theories of bottom-up and top-down determinants of the Proteus effect and thus contribute to understanding its underlying mechanisms and determinants.

This article explores the effect of one's body representation on time perception. Time Perception is modulated by a variety of factors including, e.g., the current situation or activity, it can display significant disturbances caused by psychological disorders, and it is influenced by emotional and interoceptive states, i.e., "the sense of the physiological condition of the body". We investigated this relation between one's own body and the perception of time in a novel Virtual Reality (VR) experiment explicitly fostering user activity. Forty-Eight participants randomly experienced different degrees of embodiment: i) without an avatar (low), ii) with hands (medium), and iii) with a high-quality avatar (high). Participants had to repeatedly activate a virtual lamp and estimate the duration of time intervals as well as judge the passage of time. Our results show a significant effect of embodiment on time perception: time passes slower in the low embodiment condition compared to the medium and high conditions. In contrast to prior work, the study provides missing evidence that this effect is independent of the level of activity of participants: In our task, users were prompted to repeatedly perform body actions, thereby ruling-out a potential influence of the level of activity. Importantly, duration judgements in both the millisecond and minute ranges seemed unaffected by variations in embodiment. Taken together, these results lead to a better understanding of the relationship between the body and time.

Mixed Reality (MR) applications along Milgram's Reality-Virtuality (RV) continuum motivated a number of recent theories on potential constructs and factors describing MR experiences. This paper investigates the impact of incongruencies that are processed on different information processing layers (i.e., sensation/perception and cognition layer) to provoke breaks in plausibility. It examines the effects on spatial and overall presence as prominent constructs of Virtual Reality (VR). We developed a simulated maintenance application to test virtual electrical devices. Participants performed test operations on these devices in a counterbalanced, randomized 2×2 between-subject design in either VR as congruent or Augmented Reality (AR) as incongruent on the sensation/perception layer. Cognitive incongruence was induced by the absence of traceable power outages, decoupling perceived cause and effect after activating potentially defective devices. Our results indicate that the effects of the power outages differ significantly in the perceived plausibility and spatial presence ratings between VR and AR. Both ratings decreased for the AR condition (incongruent sensation/perception) compared to VR (congruent sensation/perception) for the congruent cognitive case but increased for the incongruent cognitive case. The results are discussed and put into perspective in the scope of recent theories of MR experiences.

Multimodal Interfaces (MMIs) combining speech and spatial input have the potential to elicit minimal cognitive load. Low cognitive load increases effectiveness as well as user satisfaction and is regarded as an important aspect of intuitive use. While this potential has been extensively theorized in the research community, experiments that provide supporting observations based on functional interfaces are still scarce. In particular, there is a lack of studies comparing the commonly used Unimodal Interfaces (UMIs) with theoretically superior synergistic MMI alternatives. Yet, these studies are an essential prerequisite for generalizing results, developing practice-oriented guidelines, and ultimately exploiting the potential of MMIs in a broader range of applications. This work contributes a novel observation towards the resolution of this shortcoming in the context of the following combination of applied interaction techniques, tasks, application domain, and technology: We present a comprehensive evaluation of a synergistic speech & touch MMI and a touch-only menu-based UMI (interaction techniques) for selection and system control tasks in a digital tabletop game (application domain) on an interactive surface (technology). Cognitive load, user experience, and intuitive use are evaluated, with the former being assessed by means of the dual-task paradigm. Our experiment shows that the implemented MMI causes significantly less cognitive load and is perceived significantly more usable and intuitive than the UMI. Based on our results, we derive recommendations for the interface design of digital tabletop games on interactive surfaces. Further, we argue that our results and design recommendations are suitable to be generalized to other application domains on interactive surfaces for selection and system control tasks.

We present the gamified AR learning environment ARES. During the gameplay, learners guide the little dragon called “Ares” through dungeons. Using AR, ARES three-dimensionally integrates the dungeons in the real-world. Each dungeon represents a coordinate system with obstacles that ultimately create mathematical exercises, e.g. rotating a door to let the dragon fly through it. Overcoming such a challenge requires a learner to spatially analyze the exercise and apply the fundamental mathematical principles. ARES displays a debriefing screen at the end of a dungeon to further support the learning process. In a preliminary qualitative user study, pre-service and in-service teachers saw great potential in ARES for providing further practical and motivating exercises to deepen the knowledge in the classroom and at home.

The development of embodied Virtual Reality (VR) systems involves multiple central design choices. These design choices affect the user perception and therefore require thorough consideration. This article reports on two user studies investigating the influence of common design choices on relevant intermediate factors (sense of embodiment, presence, motivation, activation, and task load) in a VR application for physical exercises. The first study manipulated the avatar fidelity (abstract, partial body vs. anthropomorphic, full-body) and the environment (with vs. without mirror). The second study manipulated the avatar type (healthy vs. injured) and the environment type (beach vs. hospital) and, hence, the avatar-environment congruence. The full-body avatar significantly increased the sense of embodiment and decreased mental demand. Interestingly, the mirror did not influence the dependent variables. The injured avatar significantly increased the temporal demand. The beach environment significantly reduced the tense activation. On the beach, participants felt more present in the incongruent condition embodying the injured avatar.

This article investigates the effects of different XR displays on the perception and plausibility of personalized virtual humans. We compared immersive virtual reality (VR), video see-through augmented reality (VST AR), and optical see-through AR (OST AR). The personalized virtual alter egos were generated by state-of-the-art photogrammetry methods. 42 participants were repeatedly exposed to animated versions of their 3D-reconstructed virtual alter egos in each of the three XR display conditions. The reconstructed virtual alter egos were additionally modified in body weight for each repetition. We show that the display types lead to different degrees of incongruence between the renderings of the virtual humans and the presentation of the respective environmental backgrounds, leading to significant effects of perceived mismatches as part of a plausibility measurement. The device-related effects were further partly confirmed by subjective misestimations of the modified body weight and the measured spatial presence. Here, the exceedingly incongruent OST AR condition leads to the significantly highest weight misestimations as well as to the lowest perceived spatial presence. However, similar effects could not be confirmed for the affective appraisal (i.e., humanness, eeriness, or attractiveness) of the virtual humans, giving rise to the assumption that these factors might be unrelated to each other.

The embodiment of avatars in virtual reality (VR) is a promising tool for enhancing the user’s mental health. A great example is the treatment of body image disturbances, where eliciting a full-body illusion can help identify, visualize, and modulate persisting misperceptions. Augmented reality (AR) could complement recent advances in the field by incorporating real elements, such as the therapist or the user’s real body, into therapeutic scenarios. However, research on the use of AR in this context is very sparse. Therefore, we present a holographic AR mirror system based on an optical see-through (OST) device and markerless body tracking, collect valuable qualitative feedback regarding its user experience, and compare quantitative results regarding presence, embodiment, and body weight perception to similar systems using video see-through (VST) AR and VR. For our OST AR system, a total of 27 normal-weight female participants provided predominantly positive feedback on display properties (field of view, luminosity, and transparency of virtual objects), body tracking, and the perception of the avatar’s appearance and movements. In the quantitative comparison to the VST AR and VR systems, participants reported significantly lower feelings of presence, while they estimated the body weight of the generic avatar significantly higher when using our OST AR system. For virtual body ownership and agency, we found only partially significant differences. In summary, our study shows the general applicability of OST AR in the given context offering huge potential in future therapeutic scenarios. However, the comparative evaluation between OST AR, VST AR, and VR also revealed significant differences in relevant measures. Future work is mandatory to corroborate our findings and to classify the significance in a therapeutic context.

Digital reconstruction of humans has various interesting use-cases. Animated virtual humans, avatars and agents alike, are the central entities in virtual embodied human-computer and human-human encounters in social XR. Here, a faithful reconstruction of facial expressions becomes paramount due to their prominent role in non-verbal behavior and social interaction. Current XR-platforms, like Unity 3D or the Unreal Engine, integrate recent smartphone technologies to animate faces of virtual humans by facial motion capturing. Using the same technology, this article presents an optimization-based approach to generate personalized blendshapes as animation targets for facial expressions. The proposed method combines a position-based optimization with a seamless partial deformation transfer, necessary for a faithful reconstruction. Our method is fully automated and considerably outperforms existing solutions based on example-based facial rigging or deformation transfer, and overall results in a much lower reconstruction error. It also neatly integrates with recent smartphone-based reconstruction pipelines for mesh generation and automated rigging, further paving the way to a widespread application of human-like and personalized avatars and agents in various use-cases.

A novel theoretical model recently introduced coherence and plausibility as the essential conditions of XR experiences, challenging contemporary presence-oriented concepts. This article reports on two experiments validating this model, which assumes coherence activation on three layers (cognition, perception, and sensation) as the potential sources leading to a condition of plausibility and from there to other XR qualia such as presence or body ownership. The experiments introduce and utilize breaks in plausibility (in analogy to breaks in presence): We induce incoherence on the perceptual and the cognitive layer simultaneously by a simulation of object behaviors that do not conform to the laws of physics, i.e., gravity. We show that this manipulation breaks plausibility and hence confirm that it results in the desired effects in the theorized condition space but that the breaks in plausibility did not affect presence. In addition, we show that a cognitive manipulation by a storyline framing is too weak to successfully counteract the strong bottom-up inconsistencies. Both results are in line with the predictions of the recently introduced three-layer model of coherence and plausibility, which incorporates well-known top-down and bottom-up rivalries and its theorized increased independence between plausibility and presence.

The integration of Augmented Reality (AR) in teaching concepts allows for the visualization of complex learning contents and can simultaneously enhance the learning motivation. By providing Tangible Augmented Reality (TAR), an AR learning environment receives a haptic aspect and allows for a direct manipulation of augmented learning materials. However, manipulating tangible objects while using handheld AR might reduce the intuitive use and hence user experience. Users need to simultaneously control the application and manipulate the tangible object. Therefore, we compare the differences in intuitive use and user experience evoked by varied technologies of knowledge presentation in an educational context. In particular, we compare a TAR learning environment targeting the learning of the anatomy of vertebrates to its smartphone-only and AR-only versions. The three versions of the learning environment only differ in their method of knowledge presentation. The three versions show a similar perceived intuitive use. The TAR version, however, yielded a significantly higher attractiveness and stimulation than AR-only and smartphone-only. This suggests a positive effect of TAR learning environments on the overall learning experience.

This paper examines the impact of implicit and explicit feedback in Virtual Reality (VR) on performance and user experience during motor rehabilitation. In this work, explicit feedback consists of visual and auditory cues provided by a virtual trainer, compared to traditional feedback provided by a real physiotherapist. Implicit feedback was generated by the walking motion of the virtual trainer accompanying the patient during virtual walks. Here, the potential synchrony of movements between the trainer and trainee is intended to create an implicit visual affordance of motion adaption. We hypothesize that this will stimulate the activation of mirror neurons, thus fostering neuroadaptive processes. We conducted a clinical user study in a rehabilitation center employing a gait robot. We investigated the performance outcome and subjective experience of four resulting VR-supported rehabilitation conditions: with/without explicit feedback, and with/without implicit (synchronous motion) stimulation by a virtual trainer. We further included two baseline conditions reflecting the current NonVR procedure in the rehabilitation center. Our results show that additional feedback generally resulted in better patient performance, objectively assessed by the necessary applied support force of the robot. Additionally, our VR-supported rehabilitation procedure improved enjoyment and satisfaction, while no negative impacts could be observed. Implicit feedback and adapted motion synchrony by the virtual trainer led to higher mental demand, giving rise to hopes of increased neural activity and neuroadaptive stimulation.

Embodiment and body perception have become important research topics in the field of virtual reality (VR). VR is considered a particularly promising tool to support research and therapy in regard to distorted body weight perception. However, the influence of embodiment on body weight perception has yet to be clarified. To address this gap, we compared body weight perception of 56 female participants of normal weight using a VR application. They either (a) self-embodied a photorealistic, non-personalized virtual human and performed body movements in front of a virtual mirror or (b) only observed the virtual human as other's avatar (or agent) performing the same movements in front of them. Afterward, participants had to estimate the virtual human's body weight. Additionally, we considered the influence of the participants' body mass index (BMI) on the estimations and captured the participants' feelings of presence and embodiment. Participants estimated the body weight of the virtual human as their embodied self-avatars significantly lower compared to participants rating the virtual human as other's avatar. Furthermore, the estimations of body weight were significantly predicted by the participant's BMI with embodiment, but not without. Our results clearly highlight embodiment as an important factor influencing the perception of virtual humans' body weights in VR.

Historic buildings shown in open-air museums often lack a good accessibility and visitors rarely can interact with them as well as displayed tools to learn about processes. Providing these buildings in Virtual Reality could be a great supplement for museums to provide accessible and interactive offers. To investigate the effectiveness of this approach and to derive design guidelines, we developed an interactive virtual replicate of a medieval mill. We present the design of the mill and the results of a preliminary usability evaluation.

This paper presents the first prototype of a virtual audience system (VAS) specifically designed as a training tool for conference talks. This system has been tailored for university seminars dedicated to the preparation and delivery of scientific talks. We describe the required features which have been identified during the development process. We also summarize the preliminary feedback received from lecturers and students during the first deployment of the system in seminars for bachelor and doctoral students. Finally, we discuss future work and research directions. We believe our system architecture and features are providing interesting insights on the development and integration of VR-based educational tools into university curriculum.

In this paper, we adapt an existing VR framework for handwriting and sketching on physically aligned virtual surfaces to AR environments using the Microsoft HoloLens 2. We demonstrate a multimodal input metaphor to control the framework’s calibration features using hand tracking and voice commands. Our technical evaluation of fingertip/surface accuracy and precision on physical tables and walls is in line with existing measurements on comparable hardware, albeit considerably lower compared to previous work using controller-based VR devices. We discuss design considerations and the benefits of our unified input metaphor suitable for controller tracking and hand tracking systems. We encourage extensions and replication by providing a publicly available reference implementation (https://go.uniwue.de/hci-otss-hololens).

This article introduces the accessory dock, a 3D printed multi-purpose extension for consumer-grade XR controllers that enables flexible mounting of self-made and commercial accessories. The uniform design of our concept opens new opportunities for XR systems being used for more diverse purposes, e.g., researchers and practitioners could use and compare arbitrary XR controllers within their experiments while ensuring access to buttons and battery housing. As a first example, we present a stylus tip accessory to build an XR Stylus, which can be directly used with frameworks for handwriting, sketching, and UI interaction on physically aligned virtual surfaces. For new XR controllers, we provide instructions on how to adjust the accessory dock to the controller’s form factor. A video tutorial for the construction and the source files for 3D printing are publicly available for reuse, replication, and extension (https://go.uniwue.de/hci-otss-accessory-dock).

This paper presents an image-based rendering approach to accelerate rendering time of virtual scenes containing a large number of complex high poly count objects. Our approach replaces complex objects by impostors, light-weight image-based representations leveraging geometry and shading related processing costs. In contrast to their classical implementation, our impostors are specifically designed to work in Virtual-, Augmented- and Mixed Reality scenarios (XR for short), as they support stereoscopic rendering to provide correct depth perception. Motion parallax of typical head movements is compensated by using a ray marched parallax correction step. Our approach provides a dynamic run-time recreation of impostors as necessary for larger changes in view position. The dynamic run-time recreation is decoupled from the actual rendering process. Hence, its associated processing cost is therefore distributed over multiple frames. This avoids any unwanted frame drops or latency spikes even for impostors of objects with complex geometry and many polygons. In addition to the significant performance benefit, our impostors compare favorably against the original mesh representation, as geometric and textural temporal aliasing artifacts are heavily suppressed.

Multimodal Interfaces (MMIs) have been considered to provide promising interaction paradigms for Virtual Reality (VR) for some time. However, they are still far less common than unimodal interfaces (UMIs). This paper presents a summative user study comparing an MMI to a typical UMI for a design task in VR. We developed an application targeting creative 3D object manipulations, i.e., creating 3D objects and modifying typical object properties such as color or size. The associated open user task is based on the Torrence Tests of Creative Thinking. We compared a synergistic multimodal interface using speech-accompanied pointing/grabbing gestures with a more typical unimodal interface using a hierarchical radial menu to trigger actions on selected objects. Independent judges rated the creativity of the resulting products using the Consensual Assessment Technique. Additionally, we measured the creativity-promoting factors flow, usability, and presence. Our results show that the MMI performs on par with the UMI in all measurements despite its limited flexibility and reliability. These promising results demonstrate the technological maturity of MMIs and their potential to extend traditional interaction techniques in VR efficiently.

The appearance of avatars can potentially alter changes in their users’ perception and behavior. Based on this finding, approaches to support the therapy of body perception disturbances in eating or body weight disorders by mixed reality (MR) systems gain in importance. However, the methodological heterogeneity of previous research has made it difficult to assess the suitability of different MR systems for therapeutic use in these areas. The effects of MR system properties and related psychometric factors on body-related perceptions have so far remained unclear. We developed an interactive virtual mirror embodiment application to investigate the differences between an augmented reality see-through head-mounted-display (HMD) and a virtual reality HMD on the before-mentioned factors. Additionally, we considered the influence of the participant’s body-mass-index (BMI) and the BMI difference between participants and their avatars on the estimations. The 54 normal-weight female participants significantly underestimated the weight of their photorealistic, generic avatar in both conditions. Body weight estimations were significantly predicted by the participants’ BMI and the BMI difference. We also observed partially significant differences in presence and tendencies for differences in virtual body ownership between the systems. Our results offer new insights into the relationships of body weight perception in different MR environments and provide new perspectives for the development of therapeutic applications.

Immersive Virtual Reality (VR) is increasingly being explored as an alternative medium for gambling games to attract players. Typically, gambling games try to impair a player's decision making, usually for the disadvantage of the players' financial outcome. An impaired decision making results in the inability to differentiate between advantageous and disadvantageous options. We investigated if and how immersion impacts decision making using a VR-based realization of the Iowa Gambling Task (IGT) to pinpoint potential risks and effects of gambling in VR. During the IGT, subjects are challenged to draw cards from four different decks of which two are advantageous. The selections made serve as a measure of a participant's decision making during the task. In a novel user study, we compared the effects of immersion on decision making between a low-immersive desktop-3D-based IGT realization and a high immersive VR version. Our results revealed significantly more disadvantageous decisions when playing the immersive VR version. This indicates an impairing effect of immersion on simulated real life decision making and provides empirical evidence for a high risk potential of gambling games targeting immersive VR.

Latency in Virtual Reality (VR) applications can have numerous detrimental effects, e.g., a hampered user experience, a reduced user performance, or the occurrence of cybersickness. In VR environments, latency usually is measured as Motion-to-Photon (MTP) latency and reported as a mean value. This mean is taken during some specific intervals of sample runs with the target system, often detached in significant aspects from the final target scenario, to provide the necessary boundary conditions for the measurements. Additionally, the reported mean value is agnostic to dynamic and spiking latency behavior. This paper introduces an apparatus that is capable of determining per-frame MTP latency to capture dynamic MTP latency and latency jitter in addition to the commonly reported mean values of latency. The approach is evaluated by measuring MTP latency of a VR simulation based on the Unreal engine and the HTC Vive as a typical consumer-grade Head-Mounted Display (HMD). In contrast to previous approaches, the system does not rely on the HMD to be fixed to an external apparatus, can be used to assess any simulation setup, and can be extended to continuously measure latency during run-time. We evaluate the accuracy of our apparatus by injecting a controlled artificial latency in a VR simulation. We show that latency jitter artifacts already occur without system load, potentially caused by the tracking of the specific HMD, and how mean latency and jitter increase under system load, leading to dropped frames and an overall degraded system performance. The presented system can be used to monitor latency and latency jitter as critical simulation characteristics necessary to report and control to avoid unwanted effects and detrimental system performance.

Stereoscopic rendering is a prominent feature of virtual reality applications to generate depth cues and to provide depth perception in the virtual world. However, straight-forward stereo rendering methods usually are expensive since they render the scene from two eye-points which in general doubles the frame times. This is particularly problematic since virtual reality sets high requirements for real-time capabilities and image resolution. Hence, this paper presents a hybrid rendering system that combines classic rasterization and real-time ray-tracing to accelerate stereoscopic rendering. The system reprojects the pre-rendered left half of the stereo image pair into the right perspective using a forward grid warping technique and identifies resulting reprojection errors, which are then efficiently resolved by adaptive real-time ray-tracing. A final analysis shows that the system achieves a significant performance gain, has a negligible quality impact, and is suitable even for higher rendering resolutions.

The perception of time is closely related to our well-being. Psycho-pathological conditions such as depression, schizophrenia and autism are often linked to a disturbed sense of time. In this paper we present a novel framework called Metachron, which is intended to support research in the field of time perception and manipulation in Virtual Reality (VR). Our system allows the systematic modification of events in real time along the three main event axes i) Velocity, ii) Syncronicity and iii) Density. Our future work will investigate the influence of each dimension on the passage of time (varying velocity of time flow) and the structure of time (varying synchronicity of events), which should provide insights for the design of VR diagnostic and therapeutic tools.

This paper introduces an automated 3D-reconstruction method for generating high-quality virtual humans from monocular smartphone cameras. The input of our approach are two video clips, one capturing the whole body and the other providing detailed close-ups of head and face. Optical flow analysis and sharpness estimation select individual frames, from which two dense point clouds for the body and head are computed using multi-view reconstruction. Automatically detected landmarks guide the fitting of a virtual human body template to these point clouds, thereby reconstructing the geometry. A graph-cut stitching approach reconstructs a detailed texture. Our results are compared to existing low-cost monocular approaches as well as to expensive multi-camera scan rigs. We achieve visually convincing reconstructions that are almost on par with complex camera rigs while surpassing similar low-cost approaches. The generated high-quality avatars are ready to be processed, animated, and rendered by standard XR simulation and game engines such as Unreal or Unity.

This paper introduces a method for computing the difficulty of selection tasks in virtual environments using pointing metaphors by operationalizing an established human motor behavior model. In contrast to previous work, the difficulty is calculated automatically at run-time for arbitrary environments. We present and provide the implementation of our method within Unity 3D. The difficulty is computed based on a contextual analysis of spatial boundary conditions, i.e., target object size and shape, distance to the user, and occlusion. We believe our method will enable developers to build adaptive systems that automatically equip the user with the most appropriate selection technique according to the context. Further, it provides a standard metric to better evaluate and compare different selection techniques.

User embodiment is important for many virtual reality (VR) applications, for example, in the context of social interaction, therapy, training, or entertainment. However, there is no data-driven and validated instrument to empirically measure the perceptual aspects of embodiment, necessary to reliably evaluate this important phenomenon. To provide a method to assess components of virtual embodiment in a reliable and consistent fashion, we constructed a Virtual Embodiment Questionnaire (VEQ). We reviewed previous literature to identify applicable constructs and questionnaire items, and performed a confirmatory factor analysis (CFA) on the data from three experiments ( N=196). The analysis confirmed three factors: (1) ownership of a virtual body, (2) agency over a virtual body, and (3) the perceived change in the body schema. A fourth study ( N=22) was conducted to confirm the reliability and validity of the scale, by investigating the impacts of latency and latency jitter present in the simulation. We present the proposed scale and study results and discuss resulting implications.

Affine Transformations (ATs) often escape an intuitive approach due to their high complexity. Therefore, we developed GEtiT that directly encodes ATs in its game mechanics and scales the knowledge's level of abstraction. This results in an intuitive application as well as audiovisual presentation of ATs and hence in a knowledge learning. We also developed a specific Virtual Reality (VR) version to explore the effects of immersive VR on the learning outcomes. This paper presents our approach of directly encoding abstract knowledge in game mechanics, the conceptual design of GEtiT and its technical implementation. Both versions are compared in regard to their usability in a user study. The results show that both GEtiT versions induce a high degree of flow and elicit a good intuitive use. They validate the effectiveness of the design and the resulting knowledge application requirements. Participants favored GEtiT VR thus showing a potentially higher learning quality when using VR.

Virtual Reality (VR) has always been considered a promising medium to support designers with alternative work environments. Still, graphical user interfaces are prone to induce attention shifts between the user interface and the manipulated target objects which hampers the creative process. This work proposes a speech-and-gesture-based interaction paradigm for creative tasks in VR. We developed a multimodal toolbox (MTB) for VR-based design applications and compared it to a typical unimodal menu-based toolbox (UTB). The comparison uses a design-oriented use-case and measures flow, usability, and presence as relevant characteristics for a VR-based design process. The multimodal approach (1) led to a lower perceived task duration and a higher reported feeling of flow. It (2) provided a higher intuitive use and a lower mental workload while not being slower than an UTB. Finally, it (3) generated a higher feeling of presence. Overall, our results confirm significant advantages of the proposed multimodal interaction paradigm and the developed MTB for important characteristics of design processes in VR.

Gait impairments from neurological injuries require repeated and exhaustive physical exercises for rehabilitation. Prolonged physical training in clinical environments can easily become frustrating and de-motivating for various reasons which in turn risks to decrease efficiency during the healing process. This paper introduces an immersive VR system for gait rehabilitation which targets user experience and increase of motivation while evoking comparable physiological responses needed for successful training effects. The system provides a virtual environment consisting of open fields, forest, mountains, waterfalls, animals, and a beach for inspiring strolls and is able to include a virtual trainer as a companion during the walks. We evaluated the ecological validity of the system with healthy subjects before performing the clinical trial. We assessed the system's target qualities with a longitudinal study with 45 healthy participants in three consecutive days in comparison to a baseline non-VR condition. The system was able to evoke similar physiological responses. The workload was increased for the VR condition but the system also elicited a higher enjoyment and motivation which was the main goal. The latter benefits slightly decreased over time (as did workload) while they were still higher than in the non-VR condition. The virtual trainer did not show to be beneficial, the corresponding implications are discussed. Overall, the approach shows promising results which renders the system a viable alternative for the given use case while it motivates interesting direction for future work.

Slot machines are one of the most played games by pathological gamblers. New technologies, e.g. immersive Virtual Reality (VR), offer more possibilities to exploit erroneous beliefs in the context of gambling. However, the risk potential of VR-based gambling has not been researched, yet. A higher immersion might increase harmful aspects, thus making VR realizations more dangerous. Measuring harm-inducing factors reveals the risk potential of virtual gambling. In a user study, we analyze a slot machine realized as a desktop 3D and as an immersive VR version. Both versions are compared in respect to effects on dissociation, urge to gamble, dark flow, and illusion of control. Our study shows significantly higher values of dissociation, dark flow, and urge to gamble in the VR version. Presence significantly correlates with all measured harm-inducing factors. We demonstrate that VR-based gambling has a higher risk potential. This creates the importance of regulating VR-based gambling.

Biometric measures such as the electroencephalogram (EEG) promise to become viable alternatives to subjective questionnaire ratings for the evaluation of psychophysical effects associated with Virtual Reality (VR) systems, as they provide objective and continuous measurements without breaking the exposure. The extent to which the EEG signal can be disturbed by the presence of VR systems, however, has been barely investigated. This study outlines how to evaluate the compatibility of a given EEG-VR setup on the example of two commercial head-mounted displays (HMDs), the Oculus Rift and the HTC Vive Pro. We use a novel experimental protocol to compare the spectral composition between conditions with and without an HMD present during an eyes-open vs. eyes-closed task. We found general artifacts at the line hum of 50 Hz, and additional HMD refresh rate artifacts (90 Hz) for the Oculus rift exclusively. Frequency components typically most interesting to non-invasive EEG research and applications , however, remained largely unaffected. We observed similar topographies of visually-induced modulation of alpha band power for both HMD conditions in all subjects. Hence, the study introduces a necessary validation test for HMDs in combination with EEG and further promotes EEG as a potential biometric measurement method for psychophysical effects in VR systems.

Virtual Reality (VR) technology offers promising opportunities to improve traditional treadmill-based rehabilitation programs. We present an immersive VR rehabilitation system that includes a head-mounted display and motion sensors. The application is designed to promote the experience of relatedness, autonomy, and competence. The application uses procedural content generation to generate diverse landscapes. We evaluated the effect of the immersive rehabilitation system on motivation and affect. We conducted a repeated measures study with 36 healthy participants to compare the immersive program to a traditional rehabilitation program. Participants reported significant greater enjoyment, felt more competent and experienced higher decision freedom and meaningfulness in the immersive VR gait training compared to the traditional training. They experienced significantly lower physical demand, simulator sickness, and state anxiety, and felt less pressured while still perceiving a higher personal performance. We derive three design implications for future applications in gait rehabilitation: Immersive VR provides a promising augmentation for gait rehabilitation. Gamification features provide a design guideline for content creation in gait rehabilitation. Relatedness and autonomy provide critical content features in gait rehabilitation.

In this paper, we investigated the impact of an egocentric simulated frame of reference, the so-called simulated CAVE, on three type of travel techniques: Steering, Jumping and Sliding. Contrary to suggestions from previous work, no significant differences were found regarding spatial awareness between all techniques with or without the simulated CAVE. Our first results also showed a negative effect of the simulated CAVE on participants' motion sickness for every technique, while confirming that the Jumping is eliciting less motion sickness with or without it.

Technologies for Virtual, Mixed, and Augmented Reality (VR, MR, and AR) allow to artificially augment social interactions and thus to go beyond what is possible in real life. Motivations for the use of social augmentations are manifold, for example, to synthesize behavior when sensory input is missing, to provide additional affordances in shared environments, or to support inclusion and training of individuals with social communication disorders. We review and categorize augmentation approaches and propose a software architecture based on four data layers. Three components further handle the status analysis, the modification, and the blending of behaviors. We present a prototype (injectX) that supports behavior tracking (body motion, eye gaze, and facial expressions from the lower face), status analysis, decision-making, augmentation, and behavior blending in immersive interactions. Along with a critical reflection, we consider further technical and ethical aspects.

In this paper, we describe the implementation and performance of a Virtual Audience perception model for Virtual Reality (VR). The model is a VR adaptation of an existing desktop model. The system allows a user in VR to easily build and experience a wide variety of atmospheres with small or large groups of virtual agents.The paper describes results of early evaluations for this model in VR. Our first scalability benchmark results demonstrated the ability to simultaneously handle one hundred virtual agents without significantly affecting there commended frame rate for VR applications.This research is conducted in the context of a classroom simulation software for teachers’ training.

This paper presents a novel technique of navigation in Virtual Reality (VR) called Drone-Steering. This technique has been designed to facilitate path learning and traveling in VR by reducing both cybersickness and disorientation. We compared this technique to traditional Hand-Steering in a landmark-free environment. Our first experiment confirmed a significantly lower level of cybersickness during traveling and significantly better path learning. We believe that our technique constitutes a promising alternative to current VR navigation techniques, and will especially interest researchers and developers targeting large VR environments.

This article investigates the impact of waiting in Virtual Reality (VR) on the perception of time. We manipulated the visual quality of a virtual room replicating a real one (360-picture vs. 3D-model) with and without avatar embodiment (no-avatar vs. avatar). We only observed a significant difference in the estimated time duration between the real and the virtual worlds when using no avatar within a 3D model of the room. Our early results suggest that a VR environment with an avatar and a simple 3D model or 360 picture room is not significantly perturbing time perception and thus could be used for diagnosis and therapy of psychiatric conditions related to altered time perception.

This paper investigates the effects of normal mapping on the perception of geometric depth between stereoscopic and non-stereoscopic views. Results show, that in a head-tracked environment, the addition of binocular disparity has no impact on the error rate in the detection of normal-mapped geometry. It does however significantly shorten the detection time.

We explore motion parameters, more specifically gait parameters, as an objective indicator to assess simulator sickness in Virtual Reality (VR). We discuss the potential relationships between simulator sickness, immersion, and presence. We used two different camera pose (position and orientation) estimation methods for the evaluation of motion tasks in a large-scale VR environment: a simple model and an optimized model that allows for a more accurate and natural mapping of human senses. Participants performed multiple motion tasks (walking, balancing, running) in three conditions: a physical reality baseline condition, a VR condition with the simple model, and a VR condition with the optimized model. We compared these conditions with regard to the resulting sickness and gait, as well as the perceived presence in the VR conditions. The subjective measures confirmed that the optimized pose estimation model reduces simulator sickness and increases the perceived presence. The results further show that both models affect the gait parameters and simulator sickness, which is why we further investigated a classification approach that deals with non-linear correlation dependencies between gait parameters and simulator sickness. We argue that our approach could be used to assess and predict simulator sickness based on human gait parameters and we provide implications for future research.

This article investigates performance and user experience in Social Virtual Reality (SVR) targeting distributed, embodied, and immersive, face-to-face encounters. We demonstrate the close relationship between scalability, reproduction accuracy, and the resulting performance characteristics, as well as the impact of these characteristics on users co-located with larger groups of embodied virtual others. System scalability provides a variable number of co-located avatars and Al-controlled agents with a variety of different appearances, including realistic-looking virtual humans generated from photogrammetry scans. The article reports on how to meet the requirements of embodied SVR with today's technical off-the-shelf solutions and what to expect regarding features, performance, and potential limitations. Special care has been taken to achieve low latencies and sufficient frame rates necessary for reliable communication of embodied social signals. We propose a hybrid evaluation approach which coherently relates results from technical benchmarks to subjective ratings and which confirms required performance characteristics for the target scenario of larger distributed groups. A user-study reveals positive effects of an increasing number of co-located social companions on the quality of experience of virtual worlds, i.e., on presence, possibility of interaction, and co-presence. It also shows that variety in avatar/agent appearance might increase eeriness but might also stimulate an increased interest of participants about the environment.

This paper reports on the effect of visually-anchored prediction accuracy of haptic information on the perceived presence of virtual environments. We designed an experiment which explicitly prevented confounding factors potentially introduced by virtual body ownership and/or agency. The experimental design consisted of two main conditions defining congruent vs incongruent visual and haptic cues. Presence was measured during as well as after exposure. A distance estimation task solely based on motor action and the visually-anchored spatial model of the environment was executed to control for perceptual binding. 56 healthy volunteers were randomly assigned to one of two groups in a single-blind mixed-group design study. The study revealed increased presence for high prediction accuracy and decreased presence for low prediction accuracy, while perceptual binding still occurred. The observed effect sizes were in the medium range. The results indicate a significant correlation between prediction accuracy of haptic information and the perceived realness and presence of a virtual environment which gives rise to a discussion about models for dissociative symptom derealisation.

This paper presents an immersive Virtual Reality (VR) therapy system for gait rehabilitation after neurological impairments, e.g., caused by accidents or strokes: The system targets increase of patients' motivation to perform the repeated exercise by providing stimulating virtual exercise environments with the final goal to increase therapy efficiency and effectiveness. Instead of simply working out on immobile stationary devices, the system allows them to walk through and explore a stimulating virtual world. Patients are immersed in the virtual environments using a Head-Mounted Display (HMD). Walking patterns are captured by motion sensors attached to the patients' feet to synchronize locomotion speed between the real and the virtual world. A user-centered design process evaluated usability, user experience, and feasibility to confirm the overall goals of the system before any sensitive clinical trials with impaired patients can start. Overall, the results demonstrated an encouraging user experience and acceptance while it did not induce any unwanted side-effects, e.g., nausea or cyber-sickness.

This article presents an experiment exploring the possible impact of avatar's body part visibility on players' experience and performance using current Virtual Reality (VR) gaming platform capacities. In an action-based VR game, a player sees an avatar in first person perspective which is replicating his/her hand, head and body motion. In contrast to the expected outcome from non-game VR contexts, our results did not reveal significant differences with an avatar presenting an increasing number of visible body parts. The body ownership, immersion, emotional and cognitive involvements as well as the perceived control and difficulty were not improved with a more coherent virtual body. This tends to confirm the strong performance aspect of action-based games, whereby control efficiency and enemy awareness is paramount, and could overcome the perceptual, behavioural or emotional effects of avatar embodiment. Digital games are indeed prone to create an intense flow state typically reducing self-awareness, and focusing on the game completion and high performance achievements. However, further experiments with full-body tracking and different game types are necessary to confirm this trend. This research outcome motivates further analysis of the mutual influence of bottom-up and top-down factors of avatar embodiment causing psychophysical effects. In addition, it provides useful indications for VR game developers and researchers on possible effects and evaluation methods.

This paper compares teacher training in Virtual Reality (VR) to traditional approaches based on videos analysis and reflections. Our VR-assisted teacher training targets classroom management (CM) skills, using a low cost collaborative immersive VR platform. First results reveal a significant improvement using the VR approach.

This paper presents a novel approach for the augmentation of social behaviors in virtual reality (VR). We designed three visual transformations for behavioral phenomena crucial to everyday social interactions: eye contact, joint attention, and grouping. To evaluate the approach, we let users interact socially in a virtual museum using a large-scale multi-user tracking environment. Using a between-subject design (N = 125) we formed groups of five participants. Participants were represented as simplified avatars and experienced the virtual museum simultaneously, either with or without the augmentations. Our results indicate that our approach can significantly increase social presence in multi-user environments and that the augmented experience appears more thought-provoking. Furthermore, the augmentations seem also to affect the actual behavior of participants with regard to more eye contact and more focus on avatars/objects in the scene. We interpret these findings as first indicators for the potential of social augmentations to impact social perception and behavior in VR.

Low latency is a fundamental requirement for Virtual Reality (VR) systems to reduce the potential risks of cybersickness and to increase effectiveness, efficiency and user experience. In contrast to the effects of uniform latency degradation, the influence of latency jitter on user experience in VR is not well researched, although today's consumer VR systems are vulnerable in this respect. In this work we report on the impact of latency jitter on cybersickness in HMD-based VR environments. Test subjects are given a search task in Virtual Reality, provoking both head rotation and translation. One group experienced artificially added latency jitter in the tracking data of their head-mounted display. The introduced jitter pattern was a replication of a real-world latency behavior extracted and analyzed from an existing example VR-system. The effects of the introduced latency jitter were measured based on self-reports simulator sickness questionnaire (SSQ) and by taking physiological measurements. We found a significant increase in self-reported simulator sickness. We therefore argue that measure and control of latency based on average values taken at a few time intervals is not enough to assure a required timeliness behavior but that latency jitter needs to be considered when designing experiences for Virtual Reality.

Multimodal interfaces for Virtual Reality (VR), e.g., based on speech and gesture input/output (I/O), often exhibit complex system architectures. Tight couplings between the required I/O processing stages and the underlying scene representation and the simulator system's flow-of-control tend to result in high development and maintainability costs. This paper presents a maintainable solution for realizing such interfaces by means of a cherry-picking approach. A reusable multimodal I/O processing platform is combined with the simulation and rendering capabilities of the Unity game engine, allowing to exploit the game engine's superior API usability and tool support. The approach is illustrated based on the development of a multimodal VR adventure game called Space Tentacles.

This article reports the impact of the degree of personalization and individualization of users' avatars as well as the impact of the degree of immersion on typical psychophysical factors in embodied Virtual Environments. We investigated if and how virtual body ownership (including agency), presence, and emotional response are influenced depending on the specific look of users' avatars, which varied between (1) a generic hand-modeled version, (2) a generic scanned version, and (3) an individualized scanned version. The latter two were created using a state-of-the-art photogrammetry method providing a fast 3D-scan and post-process workflow. Users encountered their avatars in a virtual mirror metaphor using two VR setups that provided a varying degree of immersion, (a) a large screen surround projection (L-shape part of a CAVE) and (b) a head-mounted display (HMD). We found several significant as well as a number of notable effects. First, personalized avatars significantly increase body ownership, presence, and dominance compared to their generic counterparts, even if the latter were generated by the same photogrammetry process and hence could be valued as equal in terms of the degree of realism and graphical quality. Second, the degree of immersion significantly increases the body ownership, agency, as well as the feeling of presence. These results substantiate the value of personalized avatars resembling users' real-world appearances as well as the value of the deployed scanning process to generate avatars for VR-setups where the effect strength might be substantial, e.g., in social Virtual Reality (VR) or in medical VR-based therapies relying on embodied interfaces. Additionally, our results also strengthen the value of fully immersive setups which, today, are accessible for a variety of applications due to the widely available consumer HMDs.

We propose a new multimodal input technique for Non-critical Spontaneous Situations (NCSSs) in autonomous driving scenarios such as selecting a parking lot or picking up a hitchhiker. Speech and deictic (pointing) gestures were combined to instruct the car about desired interventions which include spatial references to the current environment (e.g., ''stop over [pointing] there'' or ''take [pointing] this parking lot''). In this way, advantages from both modalities were exploited: Speech allows for selecting from many maneuvres and functions in the car (e.g., stop, park), whereas deictic gestures provide a natural and intuitive way of indicating spatial discourse referents used in these interventions (e.g., near this tree, that parking lot). The speech and pointing gesture input was compared to speech and touch-based input in a user study with 38 participants. The touch-based input was selected as a baseline due to its widespread use in in-car touch screens. The evaluation showed that speech and pointing gestures are perceived more natural, intuitive and less cognitively demanding compared to speech and touch and are thus recommended as NCSSs intervention technique for autonomous driving.

In this paper, we present SIAM-C, an avatar-mediated communication platform to study socially immersive interaction in virtual environments. The proposed system is capable of tracking, transmitting, representing body motion, facial expressions, and voice via virtual avatars and inherits the transmission of human behaviors that are available in real-life social interactions. Users are immersed using active stereoscopic rendering projected onto a life-size projection plane, utilizing the concept of “fish tank” virtual reality (VR). Our prototype connects two separate rooms and allows for socially immersive avatar-mediated communication in VR.

In this paper we present a complete pipeline to create ready-to-animate virtual humans by fitting a template character to a point set obtained by scanning a real person using multi-view stereo reconstruction. Our virtual humans are built upon a holistic character model and feature a detailed skeleton, fingers, eyes, teeth, and a rich set of facial blendshapes. Furthermore, due to the careful selection of techniques and technology, our reconstructed humans are quite realistic in terms of both geometry and texture. Since we represent our models as single-layer triangle meshes and animate them through standard skeleton-based skinning and facial blendshapes, our characters can be used in standard VR engines out of the box. By optimizing for computation time and minimizing manual intervention, our reconstruction pipeline is capable of processing whole characters in less than ten minutes.

This paper investigates the effect of avatar realism on embodiment and social interactions in Virtual Reality (VR). We compared abstract avatar representations based on a wooden mannequin with high fidelity avatars generated from photogrammetry 3D scan methods. Both avatar representations were alternately applied to participating users and to the virtual counterpart in dyadic social encounters to examine the impact of avatar realism on self-embodiment and social interaction quality. Users were immersed in a virtual room via a head mounted display (HMD). Their full-body movements were tracked and mapped to respective movements of their avatars. Embodiment was induced by presenting the users' avatars to themselves in a virtual mirror. Afterwards they had to react to a non-verbal behavior of a virtual interaction partner they encountered in the virtual space. Several measures were taken to analyze the effect of the appearance of the users' avatars as well as the effect of the appearance of the others' avatars on the users. The realistic avatars were rated significantly more human-like when used as avatars for the others and evoked a stronger acceptance in terms of virtual body ownership (VBO). There also was some indication of a potential uncanny valley. Additionally, there was an indication that the appearance of the others' avatars impacts the self-perception of the users.

Modularity, modifiability, reusability, and API usability are important software qualities that determine the maintainability of software architectures. Virtual, Augmented, and Mixed Reality (VR, AR, MR) systems, modern computer games, as well as interactive human-robot systems often include various dedicated input-, output-, and processing subsystems. These subsystems collectively maintain a real-time simulation of a coherent application state. The resulting interdependencies between individual state representations, mutual state access, overall synchronization, and flow of control implies a conceptual close coupling whereas software quality asks for a decoupling to develop maintainable solutions. This article presents five semantics-based software techniques that address this contradiction: Semantic grounding, code from semantics, grounded actions, semantic queries, and decoupling by semantics. These techniques are applied to extend the well-established entity-component-system (ECS) pattern to overcome some of this pattern's deficits with respect to the implied state access. A walk-through of central implementation aspects of a multimodal (speech and gesture) VR-interface is used to highlight the techniques' benefits. This use-case is chosen as a prototypical example of complex architectures with multiple interacting subsystems found in many VR, AR and MR architectures. Finally, implementation hints are given, lessons learned regarding maintainability pointed-out, and performance implications discussed.

Classical reminiscence therapy has been shown to effectively enhance the stability of memory and identity in people with dementia. Typically, reminiscence therapy uses biography artifacts like photos and personal items and objects. Today, many of these artifacts are from the digital realm providing new options to adapt or even improve the purely analog therapy. In this work we propose a method to enhance reminiscence therapy by computer simulated biographic associations. Our approach provides assistance for associative reasoning on affective stimuli and thus enables access to biographic content so that no deliberate search is required. We develop a recommender model for mapping mental states to biographic content based on similarity. The system dynamically adapts its state and the depicted digital artifacts to the responses of the user. It is a first step towards an immersive reminiscence therapy which will incorporate associated stimuli on multiple channels to increase effectiveness. A preliminary study showed encouraging results concerning the usability of the system.

This paper presents a mixed reality tabletop role-playing game with a novel combination of interaction styles and gameplay mechanics. Our contribution extends previous approaches by abandoning the traditional turn-based gameplay in favor of simultaneous real-time interaction. The increased cognitive and physical load during the simultaneous control of multiple game characters is counteracted by two features: First, certain game characters are equipped with AI-driven capabilities to become semi-autonomous virtual agents. Second, (groups of) these agents can be instructed by high-level commands via a multimodal - speech and gesture - interface.

This paper introduces a low-cost framework capable of combining both real-time markerless face and body tracking for faithful avatar embodiment in Virtual Reality (VR). We discuss suitable hardware and software solutions and present a first prototype. This work lays the technological basis for further research on the importance of the appearance and behavioral realism of avatars, e.g., for the illusion of virtual body ownership, for social interactions in VR, as well as for VR entertainment applications (immersive games or movies).

This article introduces model checking as an alternative method to estimate the latency and parallelism of asynchronous Realtime Interactive Systems (RISs). Five typical concurrency and synchronization schemes often found in concurrent Virtual Reality (VR) and computer game systems are identified as use-cases. These use-cases guide the development a) of software primitives necessary for the use-case implementation based on asynchronous RIS architectures and b) of a graphical editor for the specification of various concurrency and synchronization schemes (including the use-cases) based on these primitives. Several model-checking tools are evaluated against typical requirements in the RIS area. As a result, the formal model checking language Rebeca and its model checker RMC are applied to the specification of the use-cases to estimate latency and parallelism for each case. The estimations are compared to measured results achieved by classical profiling from a real-world application. The estimated results of the latencies by model checking approximated the measured results adequately with a minimal difference of 3.9% in the best case and -26.8% in the worst case. It also detected a problematic execution path not covered by the stochastic nature of the measured profiling samples. The estimated results of the degree of parallelization by model checking are approximated with an minimal difference of 9.3% and a maximal difference of -28.8%. Finally, the effort of model checking is compared to the effort of implementing and profiling a RIS.

In this paper, we compare a full body marker set with a reduced rigid body marker set supported by inverse kinematics. We measured system latency, illusion of virtual body ownership, and task load in an applied scenario for inducing acrophobia. While not showing a significant change in body ownership or task performance, results do show that latency and task load are reduced when using the rigid body inverse kinematics solution. The approach therefore has the potential to improve virtual reality experiences.

In this paper, we describe an experimental method to investigate the effects of reduced social information and behavioral channels in immersive virtual environments with full-body avatar embodiment. We compared physical-based and verbal-based social interactions in real world (RW) and virtual reality (VR). Participants were represented by abstract avatars that did not display gaze, facial expressions or social cues from appearance. Our results show significant differences in terms of presence and physical performance. However, differences in effectiveness in the verbal task were not present. Participants appear to efficiently compensate for missing social and behavioral cues by shifting their attentions to other behavioral channels.

Latency jitter is a pressing problem in Virtual Reality (VR) applications. This paper analyzes latency jitter caused by typical interprocess communication (IPC) techniques commonly found in today's computer systems used for VR. Test programs measure the scalability and latencies for various IPC techniques, where increasing number of threads are performing the same task concurrently. We use four different implementations on a vanilla Linux kernel as well as on a real-time (RT) Linux kernel to further assess if a RT variant of a multiuser multiprocess operating system can prevent latency spikes and how this behavior would apply to different programming languages and IPC techniques. We found that Linux RT can limit the latency jitter at the cost of throughput for certain implementations. Further, coarse grained concurrency should be employed to avoid adding up of scheduler latencies, especially for native system space IPC, while actor systems are found to support a higher degree of concurrency granularity and a higher level of abstraction.

We present four raycast-based techniques that determine the transformation between a depth camera's coordinate system and the coordinate system defined by a rectangular surface. In addition, the surface's dimensions are measured. In contrast to other approaches, these techniques limit additional hardware requirements to commonly available, low-cost artifacts and focus on simple non-laborious procedures. A preliminary study examining our Kinect v2-based proof of concept revealed promising first results. The utilized software is available as an open-source project.

Remote maintenance of industrial manipulators often is performed via telephone support. Recent approaches in the context of the 'Industry 4.0' consider internet technologies and Augmented Reality (AR) to enhance situation awareness between external experts and local service technicians. We present two AR-based case studies: First, a mobile AR architecture based on optical see through glasses is used for an on-site local repair task. Second, a remote architecture based on a portable tablet PC and a high precision tracking system is used to realize an off-site expert access. The to-be-serviced machine is visualized inside of a large area similar to a machinery hall and can be inspected by the experts walking around this virtual plant using the tablet and perspectively correct rendering to understand the production process and the operation context. Both methods have been evaluated in first user studies.

This paper introduces an interactive surface concept for Mixed Reality (MR) tabletop games that combines a variable (LCD and/or projection) screen configuration with the detection of finger touches, in-air gestures, and tangibles. It is low-cost and minimally requires an ordinary table, a TV screen, and a Kinect v2 sensor. Existing applications can easily be connected by being compliant to standards. The concept is intended to foster further research on collaborative tabletop situations, not limited to games, but also including learning, meetings, and social interaction.

This article presents a fully immersive portable low-cost Virtual Reality system to train classroom management skills. An instructor controls the simulation of a virtual classroom populated with 24 semi-autonomous virtual agents via a desktop-based graphical user interface (GUI). The GUI provides behavior control and trainee evaluation widgets alongside a non-immersive view of the class and the trainee. The trainee's interface uses an Head-Mounted Display (HMD) and earphones for output. A depth camera and the HMD's built-in motion sensors are used for tracking the trainee and for avatar animation. An initial evaluation of both interfaces confirms the system's usefulness, specifically its capability to successfully simulate critical aspects of classroom management.

This paper introduces a fake mirror system as a research tool to study the effect of avatar embodiment with non-visually immersive virtual environments. The system combines marker-less face and body tracking to animate the individual avatars seen in a stereoscopic display with a correct perspective projection. The display dimensions match typical dimensions of a real physical mirror and the animated avatars are rendered based on a geometrically correct reflection as expected from a real mirror including correct body and face animations. The first evaluation of the system reveals the high acceptance of the setup as well as a convincing illusion of a real mirror with different types of avatars.

This paper presents an exploratory experiment measuring the role of audio feedback on the illusion of virtual body ownership (IVBO) under non-immersive mixed reality (MR) settings with Human and Non-Human avatars. Our preliminary results revealed that all avatars elicited a similar level of IVBO, despite the addition of audio feedback.

In this paper, we investigate the impact of avatar anthropomorphism on the fear of heights, when using full body avatar embodiment under an immersive virtual reality (VR) setting. Clear differences could be found in perceived anthropomorphism, but preliminary results do not show differences in stress level between Human and Non-Human avatars, although a high level of perceived secureness was reported with Non-Human avatars.

This article presents the Gamified Training Environment for Affine Transformations (GEtiT). GEtiT uses a 3D environment to visualize the effects of object rotation, translation, scaling, reflection, and shearing in 3D space. It encodes the abstract knowledge about homogeneous transformations and their order of application using specific game mechanics encoding 3D movements on different levels of abstraction. Progress in the game requires mastering of the game mechanics of a certain level of abstraction to modify objects in 3D space to a desired goal position and/or shape. Each level increases the abstraction of the representation towards a final 4 × 4 homogeneous matrix representation. Executing the game mechanics during the gameplay results in an effective training of knowledge due to a constant repetition. Evaluation showed a learning effect that is equal to a traditional training method while it achieved a higher enjoyment of use indicating that the learning quality was superior to the traditional training method.

In this paper, we present a avatar-mediated communication platform for socially immersive interaction in virtual reality (VR). Our approach is based on the combination of body tracking, facial expression tracking and "fishtank" VR. Our prototype enables two remote users to communicate via avatars.

A low latency is a fundamental timeliness requirement to reduce the potential risks of cyber sickness and to increase effectiveness, efficiency, and user experience of Virtual Reality Systems. The effects of uniform latency degradation based on mean or worst-case values are well researched. In contrast, the effects of latency jitter, the distribution pattern of latency changes over time has largely been ignored so far although today's consumer VR systems are extremely vulnerable in this respect. We investigate the applicability of the Walsh, generalized ESD, and the modified z-score test for the detection of outliers as one central latency distribution aspect. The tests are applied to well defined test cases mimicking typical timing behavior expected from concurrent architectures of today. We introduce accompanying graphical visualization methods to inspect, analyze and communicate the latency behavior of VR systems beyond simple mean or worst-case values. As a result, we propose a stacked modified z-score test for more detailed analysis.

This poster presents a maintainable method to manage lexical information required for multimodal interfaces. It is tailored for the application in real-time interactive systems, specifically for Virtual Reality, and solves three problems commonly encountered in this context: (1) The lexical information is defined on and grounded in a common knowledge representation layer (KRL) based on OWL. The KRL describes application objects and possible system functions in one place and avoids error-prone redundant data management. (2) The KRL is tightly integrated into the simulator platform using a semantically enriched object model that is auto-generated from the KRL and thus fosters high performance access. (3) A well-defined interface provides application wide access to semantic application state information in general and the lexical information in specific, which greatly contributes to decoupling, maintainability, and reusability.

Today's physics engines mainly simulate classical mechanics and rigid body dynamics, with some late advances also capable of simulating massive particle systems and some approximations of fluid dynamics. An accurate numerical simulation of complex nonmechanical processes in real time is beyond the state of the art in the respective fields. This paper illustrates an alternative approach to a purely numerical solution. It uses a semantic representation of physical properties and processes as well as a reasoning engine to model cause and effect between objects, based on their material properties. Classical collision detection is combined with semantic rules to model various physical processes, for example, in the areas of thermodynamics, electrodynamics, and fluid dynamics as well as chemical processes. Each process is broken down into fine-grained subprocesses capable of approximating continuous transitions with discretized state changes. Our system applies these high-level state descriptions to low-level value changes, which are directly mapped to a graphical representation of the scene. We demonstrate our framework's ability to support multiple complex, causally connected physical and chemical processes by simulating a Goldberg machine. Our performance benchmarks validate its scalability and potential application for entertainment or edutainment purposes.

In this paper we present a preliminary study of the impact of avatar realism on illusion of virtual body ownership (IVBO), when using a full body virtual mirror for fitness training. We evaluated three main types of user representation: realistic and non-realistic avatars as well as no avatar at all. Our results revealed that same-gender realistic human avatar elicited a slightly higher level of illusion and performance. However qualitative analysis of open questions revealed that the feeling of power was higher with non-realistic strong-looking avatars.

In this paper we present a novel experiment to explore the impact of avatar realism on the illusion of virtual body ownership (IVBO) in immersive virtual environments, with full-body avatar embodiment and freedom of movement. We evaluated four distinct avatars (a humanoid robot, a block-man, and both male and female human adult) presenting an increasing level of anthropomorphism in their detailed compositions Our results revealed that each avatar elicited a relatively high level of illusion. However both machine-like and cartoon-like avatars elicited an equivalent IVBO, slightly superior to the human-ones. A realistic human appearance is therefore not a critical top-down factor of IVBO, and could lead to an Uncanney Valley effect.

In this paper we present a study of the impact of avatar realism on user experience and performance in stressful immersive virtual environments. We evaluated a stressful and a stress-free environment with partial avatar embodiment under low (iconic) or high (photorealistic) visual fidelity conditions. An experiment with forty participants did not reveal any significant differences between both graphical versions. This first result represents an interesting finding since non realistic avatar and environment representations are faster and more economical to produce while requiring less computational resources.

Quest — XRoads is a multimodal and multimedia mixed reality version of the traditional role-play tabletop game Quest: Zeit der Helden. The original game concept is augmented with virtual content to permit a novel gaming experience. The demonstration consists of a turn-based skirmish, where players have to collaborate, control heroes or villains and use their abilities via speech, gesture, touch as well as tangible interactions.

This article describes a set of metrics for a message-passing-based asynchronous Realtime Interactive System (RIS). Current trends in concurrent RISs are analyzed, several profiling tools are outlined, and common metrics are identified. A set of nine metrics is presented in a unified and formalized way. The implementation of a profiler that measures and calculates these metrics is illustrated. The implementation of an instrumentation and a visualization tool are described. A case study shows how this approach proved beneficial during the optimization of latency of an actual system.

Precise, accurate, fast, and low-latency motion tracking is a core requirement of real-time human-computer interfaces. The choices of tracking systems for a particular set of 3D interaction techniques are manifold. Hence, guidance in this task is greatly beneficial. In this paper, we propose to establish a set of canonical and simple game-based benchmarks for a potentially standardised comparison of tracking systems. The benchmarks focus on usability scores given reoccurring interaction tasks without requiring potentially missing, incomplete, or complex latency or accuracy raw measurements. Our first two benchmarks evaluate three tracking systems regarding motion-parallax and 3D object manipulation techniques. Our usability comparisons confirmed an expected advantage of low-latency/high-accuracy systems, while they also demonstrated that certain tracking systems perform better than suggested by previous measurements of their raw performances. This indicates that our approach provides an adequate replacement and improvement over the pure comparison of technical specifications. We believe our benchmarks could benefit the research community by facilitating a usability-based comparison of motion tracking systems.

This article evaluates the utility of three technical design approaches implemented during the development of a Realtime Interactive Systems (RIS) architecture focusing on the areas of Virtual and Augmented Reality (VR and AR), Robotics, and Human-Computer Interaction (HCI). The design decisions are (1) the choice of the Scala programming language, (2) the implementation of the actor computational model, and (3) the central incorporation of ontologies as a base for semantic modeling, required for several Artificial Intelligence (AI) methods. A white-box expert review is applied to a detailed use case illustrating an interactive and multimodal game scenario, which requires a number of complex functional features like speech and gesture processing and instruction mapping. The review matches the three design decisions against three comprehensive non-functional requirements from software engineering: Reusability, scalability, and extensibility. The qualitative evaluation is condensed to a semi-quantitative summary, pointing out the benefits of the chosen technical design.

We present SiXton 's Curse-a computer game-to illustrate the ben efits of a novel simulation platform. Simulator X [2] targets vir tual, augmented, and mixed reality applications as well as computer games. The game simulates a medieval village called SiXton that can be explored and experienced using gestures and speech for in put. SiXton's Curse utilizes multiple independent components for physical simulation, sound and graphics rendering, artificial intelli gence, as well as for multi-modal interaction (MMI). The compo nents are already an integral part of Simulator X's current version. Building on Hewitt's actor model, the Simulator X platform enables the developer to easily exploit the capabilities of modern hardware architectures. A state variable concept is implemented on top of the actor model to grant uniform and easy access to global states and values by using the internal mechanisms of the actor model. Communication via an asynchronous messaging in terface reduces component coupling. The scalability of the actor model provides a uniform concurrency paradigm on different levels of granularity as well as exchangeability of architectural elements and components.

This article presents a platform for software technology research in the area of intelligent Realtime Interactive Systems. Simulator X is targeted at Virtual Reality, Augmented Reality, Mixed Reality, and computer games. It provides a foundation and testbed for a variety of different application models. The current research architecture is based on the actor model to support fine grained concurrency and parallelism. Its design follows the minimize coupling and maximize cohesion software engineering principle. A distributed world state and execution scheme is combined with an object-centered world view based on an entity model. Entities conceptually aggregate properties internally represented by state variables. An asynchronous event mechanism allows intra- and interprocess communication between the simulation actors. An extensible world interface uses an ontology-based semantic annotation layer to provide a coherent world view of the resulting distributed world state and execution scheme to application developers. The world interface greatly simplifies configurability and the semantic layer provides a solid foundation for the integration of different Artificial Intelligence components. The current architecture is implemented in Scala using the Java virtual machine. This choice additionally fosters low-level scalability, portability, and reusability.

Interaction in conversational interfaces strongly relies on the system's capability to interpret the user's references to objects via deictic expressions. Deictic gestures, especially pointing gestures, provide a powerful way of referring to objects and places, e.g., when communicating with an embodied conversational agent in a virtual reality environment. We highlight results drawn from a study on pointing and draw conclusions for the implementation of pointing-based conversational interactions in partly immersive virtual reality.

Engineering systems plays a central role in the development of successful Virtual Reality (VR) and Augmented Reality (AR) applications. Increasing computational resources are utilized to build increasingly complex artificial environments and extensive Human-Computer Interaction (HCI) systems. These types of Realtime Interactive Systems (RIS) establish a closed HCI loop. They are characterized as systems continuously analyzing users' input while concurrently synthesizing appropriate output for several of the human senses in real-time.

We introduce semantic reflection as an architectural concept for intelligent virtual environments (IVEs). SCIVE, a dedicated IVE simulation core, combines modularity with close coupled integrative aspects to provide semantic reflection on multiple layers from low-level simulation core logic, specific simulation modules' application definitions, to high-level semantic environment descriptions. SCIVE's knowledge representation layer provides the central organizing structure which ties together data representations of simulation modules, e.g., for graphics, physics, audio, haptics, or AI etc., while it additionally allows bidirectional knowledge driven access between the modules

All in-text\treferences\tunderlined\tin\tblue\tare\tlinked\tto\tpublications\ton\tResearchGate, letting you\taccess\tand\tread\tthem\timmediately.

Advanced VR simulation systems are composed of several components with independent and heterogeneously structured databases. To guarantee a closed and consistent world simulation, flexible and robust data exchange between these components has to be realized. This multiple database problem is well known in many distributed application domains, but it is central for VR setups composed of diverse simulation components. Particularly complicated is the exchange between object-centered and graph-based representation formats, where entity attributes may be distributed over the graph structure. This article presents an abstract declarative attribute representation concept, which handles different representation formats uniformly and enables automatic data exchange and synchronization between them. This mechanism is tailored to support the integration of a central knowledge component, which provides a uniform representation of the accumulated knowledge of the several simulation components involved. This component handles the incoming-possibly conflicting-world changes propagated by the diverse components. It becomes the central instance for process flow synchronization of several autonomous evaluation loops.

This article describes an integration of knowledge based techniques into simulative virtual reality (VR) applications motivated using a virtual construction task. An abstract knowledge representation layer (KRL) provides a base formalism for the integration of simulation semantics. The KRL approach is demonstrated using a generalized scene graph representation which introduces an abstract definition and implementation of geometric node interrelations.

This article presents one core component for enabling multimodal-speech and gesture-driven interaction in and for virtual environments. A so-called temporal Augmented Transition Network (tATN) is introduced. It allows to integrate and evaluate information from speech, gesture, and a given application context using a combined syntactic/semantic parse approach. This tATN represents the target structure for a multimodal integration markup language (MIML). MIML centers around the specification of multimodal interactions by letting an application designer declare temporal and semantic relations between given input utterance percepts and certain application states in a declarative and portable manner. A subsequent parse pass translates MIML into corresponding tATNs which are directly loaded and executed by a simulation engines scripting facility.