In Mixed Reality (MR), in-the-moment guidance can support physical tasks under time and attention constraints. In contexts where such guidance is embodied, users must rapidly evaluate whether to trust and follow an advisor’s recommendations. Prior work has examined how avatar representation shapes trust and social perception, but less is known about user responses when guidance conflicts and multiple embodied agents compete for attention in real time. We study trust-relevant perception and behavior in an MR paradigm where two co-present agents represent the same advisor and deliver step-by-step instructions with predefined disagreements. In a within-subjects study, we compared a volumetric avatar (VA) and a high-fidelity 3D character avatar (CA), holding identity and expressive behavior constant, and measured trust, social presence, and behavioral adherence during contradictory steps. Rendering style produced no detectable differences in trust, social presence, or co-presence. Instead, participants managed disagreement through mixed adherence and switching between agents, suggesting that instruction competition and time pressure strongly shape decision-making. These findings position conflicting guidance as a distinct context for studying trust in embodied agents, where compliance may reflect rapid prioritization rather than stable commitment to a representation. We discuss implications for designing and evaluating MR agents when timing and attention constraints can overshadow representation cues.

Video pass-through (VPT) techniques often cause visual distortions like the ‘telescope eye' effect, which leads to less immersion and spatial awareness in Extended Reality (XR). This study investigates the potential of 3D Gaussian Splatting (3DGS) as a rendering alternative to video pass-through, particularly focusing on its ability to mitigate common perceptual artifacts and enhance immersion. To evaluate the efficacy of 3DGS in an XR context, we conducted a user study comparing two visualization types: (1) conventional VPT and (2) 3DGS-based scene representation. Three distinct scenes with high detail, natural depth perception, and reflective surfaces were included in the study. An empirical study was conducted with a structured questionnaire that inquired about the level of immersion and the impact of the visual stimuli. Our results provide insights into the scalability and hardware considerations for implementing 3DGS in realtime interactive environments. The evaluation results indicated that users with a 3DGS visualization experienced a reduction in visual distortion and an enhancement in the perceived realism of objects. The findings show that 3DGS can serve as an effective alternative to VPT, potentially improving realism and reducing perceptual artifacts in XR applications.

Current and, above all, future mixed reality services and systems collect a large amount of different personal and environmental data of the users and other involved persons. Current legal regulations on data protection and governance do not appear to adequately meet these challenges. This paper looks at data collection in Mixed Reality systems in order to create a greater awareness for its far-reaching privacy-related implications. We present a straightforward scenario where a user named Melanie interacts with both her environment and another user. It is stipulated that Melanie is not a first-time user, and all interactions and environmental factors are predetermined by an existing data profile. This paper examines the technical design of the scenario, the types of data collected, and their legal implications. To address the far-reaching privacy implications and shortcomings in the current legal framework, we propose technical solutions for less privacy-encroaching data uses and recommend the implementation of better legal instruments and safeguards in the future. In conclusion we suggest that better legal instruments and safeguards are needed in future.

Abstract Behavioral realism plays a crucial role in virtual and augmented reality (VR/AR). Various avatar animation techniques, ranging from full-body motion capture to single-point inverse kinematics (IK), offer different levels of realism. While the animation of a user’s own avatar influences embodiment, the perceived realism of others’ avatars is equally important for immersion. This study ( N = 53) examines how users in smartphone AR, head-mounted display (HMD) AR, and VR perceive the behavioral realism of avatars animated with single-point IK compared to those driven by full-body motion capture. In addition, we explore whether the congruence between visual fidelity of an avatar and tracking accuracy affects perception. Our findings indicate that full-body motion capture produces significantly higher perceived realism than single-point IK, but the type of device does not have measurable impact. Furthermore, while congruence between visual realism and tracking fidelity was expected to play a role, our results suggest that its influence is limited. Despite lower realism than motion capture, modern IK techniques are still perceived positively, highlighting their viability for multi-user AR and VR applications.

Abstract To achieve realistic lighting in augmented reality (AR), virtual reality (VR), and similar applications especially with stringent realtime constraints, environmental lighting is often captured and stored in light probes, which are strategically placed throughout the scene. This study explores the optimal positioning of light probes to enhance the realism of extended realities. A novel spatial geometry-based method for light probe placement is proposed and compared against three alternative techniques. They are evaluated across various 3D Gaussian splatting environments with different probe distributions. The techniques are compared to ground truth renderings that are obtained using a pipeline that combines fast Gaussian rasterization with ray tracing. The findings demonstrate that the careful placement of a minimal number of probes can effectively replicate realistic lighting in complex environments. The geometry-based optimization method outperforms the alternatives in indoor environments by providing an efficient realtime solution with fewer probes. This optimized approach is suitable for direct implementation in realtime applications such as AR, VR, game development, and virtual productions, offering both improved lighting realism and optimized computational efficiency.

Behavioral realism is a key element in social virtual and augmented reality (VR/AR). Different techniques, from full-body to single-point IK, are used to animate virtual humans, providing varying levels of behavioral realism. While the animation method of the user’s own avatar is important, e.g., considering embodiment, the animation of others is equally important to display a convincing immersive experience. This study ($\mathrm{N}=36$) investigates how users (in AR or VR) rate the behavioral realism of others when the other person’s avatar is animated using a single point Inverse Kinematics (IK) and compares results to a full-body motion-capture-driven avatar. In addition, we investigate whether congruence between visual realism and tracking quality matters. Our results reveal that animations based on full-body motion capture are found to result in higher perceived behavioral realism compared to those using single-point inverse kinematics (IK), that device type has no influence, and that congruence seems to matter less. Despite not reaching the same level of realism as motion-captured animations, our results suggest the applicability of state-of-the-art IK techniques for multi-user applications in AR and VR as, in general, participants rated both positively.

In order to achieve realistic lighting in augmented reality (AR), virtual reality (VR), and other virtual environments with stringent real-time requirements, environmental lighting is captured and stored in light probes that are strategically positioned throughout these environments. This study investigates the optimal positioning of light probes based on the environment to achieve realistic virtual representations. A novel spatial geometry-based approach for light probe placement is introduced, validated in a comparative analyses against three alternative methods across diverse virtual environments, each utilizing different probe quantities. The investigation reveals that strategic deployment of a minimal number of probes can effectively simulate realistic illumination across complex environmental scenarios. The geometry-based optimization outperforms alternative methods by achieving efficient real-time probe placement with minimal number of probes. This optimized methodology can be directly implemented at runtime in various applications, including AR, VR, game development, and virtual productions, thereby enhancing lighting realism while optimizing computational resources.

In unusual environments and situations like extreme sports, underground environments, underwater, conflict zones or outer space, human sensory perception is often limited, which can adversely affect task performance, situational awareness, and the feeling of presence. This paper explores the efficacy of augmenting human perception with audiovisual information to support individuals in such contexts. Specifically, we investigate the impact of audiovisual augmentation on presence, situational awareness, and task performance. To conduct this research, we designed a virtual reality (VR) simulation of a space mission aboard the International Space Station (ISS). Within this simulation, participants were tasked with performing maintenance activities while receiving artificial augmentations. We conducted a user study involving 43 participants who performed the same maintenance task under four augmentation conditions: audio cues, visual cues, audiovisual cues, and no cues. Our findings reveal that adding audiovisual information significantly enhances performance. Participants with audiovisual augmentations had a 60% improvement in task completion speed compared to those without augmentations. Moreover, the workload was substantially reduced, and the sense of presence was increased when audiovisual support was used. The results highlight the potential of audiovisual augmentation as a valuable tool for individuals engaging in situations with reduced audiovisual perception. The insights demonstrating the versatility and promise of audiovisual augmentation in enhancing task performance.

Given the profound impact of staying socially connected on the well-being of older adults, this study explores the potential of augmented reality (AR) systems to enrich their social lives. A wearable AR communication system prototype was developed and tested in a user study involving N = 16 older adults from Germany. Participants wore an AR headset and engaged in a conversation task with a remote person represented by an avatar. Older adults’ experiences were assessed using think-aloud protocols, qualitative observations, posttest questionnaires, and semi-structured oral interviews. Preliminary findings indicate overall participant satisfaction, with minimal observed difficulties in headset usage and avatar-mediated interpersonal communication. The positive engagement during AR conversations highlights the system’s potential to provide positive communication experiences among older individuals. This work-in-progress paper introduces the developed system prototype and outlines the conducted user study. Further data analyses will provide deeper insights into older adults’ experiences with the system. The results will contribute to refining the prototype and offer valuable insights for the development of AR communication systems tailored to the needs and preferences of older adults.

Augmented reality (AR) presents vivid opportunities for interpersonal communication. With the growing diversity of social AR users, understanding their unique needs and perceptions becomes crucial. This study delves into how younger, middle-aged, and older adults perceive avatars with different aging attributes and degrees of realism, focusing on their anticipated user experience within a social AR system. We conducted an online within-subjects experiment involving $N = 2086$ age-diverse participants from Germany who assessed a set of nine gender-matched avatars for their perceived social attractiveness (research question 1 = RQ1) and the likelihood of selecting these avatars for self-representation in social AR (RQ2). The evaluated avatars represented different age groups (younger, middle-aged, and older) and levels of realism (low, medium, and high). We validated both the created avatars and our experimental setup and employed a linear mixed-effects modeling approach to analyze the data. Our findings unveiled a strong preference for younger high-realism avatars as communication partners (RQ1), which was consistent across all participant age groups. Similarly, participants favored younger high-realism avatars for self-representation in social AR (RQ2). However, older adults were more inclined to opt for avatars resembling their actual age. The study highlights the prevalence of age-related stereotypes in avatar-based communication. Similar to face-to-face social interactions, these stereotypes tend to render older avatars less socially attractive than their younger counterparts, irrespective of the avatar’s degree of realism. Our results invite considerations on how to combat these stereotypes through a more thoughtful and inclusive avatar design process that encompasses a broader spectrum of aging attributes.

Realistic lighting approaches typically rely on physically-based rendering which in turn often makes use of image-based lighting. Enabling these techniques in augmented reality on mobile devices requires unique approaches to estimate light probes, given the limited camera and sensor data available. In this paper, we evaluate different time-dependent and time-independent techniques for light probe estimation in augmented reality applications that try to predict the environment lighting using single images or video streams in combination with inpainting techniques. We simulate real-world applications using an evaluation framework where a simulated mobile device captures camera streams from different scenarios following a pre-defined path. The resulting camera streams are fed to a total of six estimation techniques in order to create light probes which are then used to render virtual objects while applying various materials. By comparing the rendered images as well as the light probe estimations, we perform a quantitative evaluation. We show how approaches that are able to process continuous video streams can provide more plausible results in cases where sufficient camera movement is present. Additionally, we investigate the visual impression of different types of materials showing that rough surfaces with distinct colors are less likely to produce divergent estimation results.

360-degree image data is a crucial aspect in graphics applications where they are typically used for lighting purposes. Fields like mixed reality generally rely on lighting estimation techniques to estimate the 360-degree environment. To evaluate such approaches, accurate image assessment in this domain is important. However, traditional image evaluation metrics like SSIM, PSNR, and IMED are problematic when analyzing 360-degree image data as they would require two-dimensional representations like equirectangular panoramas or unfolded cubes that introduce image distortions. In this paper, we address this problem by presenting Cube-SSIM, a variant of SSIM designed specifically for cube maps. For this, we modify SSIM to take the solid angles of cube map pixels into account which gives more consistent results than using SSIM for the individual cube faces. The computations can run efficiently on graphics hardware due to their native support for cube maps and no further image conversions are required. We show that our approach allows for more accurate results than other comparison metrics that largely depend on 2D images. While SSIM is especially important due to its wide usage, the modification can also be applied to other image metrics for which we include IMED as an example.

Abstract In recent years, various depth sensors that are small enough to be used with mobile hardware have been introduced. They provide important information for use cases like 3D reconstruction or in the context of augmented reality where tracking and camera data alone would be insufficient. However, depth sensors may not always be available due to hardware limitations or when simulating augmented reality applications for prototyping purposes. In these cases, different approaches like stereo matching or depth estimation using neural networks may provide a viable alternative. In this paper, we therefore explore the imitation of depth sensors using deep neural networks. For this, we use a state-of-the-art network for depth estimation and adapt it in order to mimic a Structure Sensor as well as an iPad LiDAR sensor. We evaluate the network which was pre-trained on NYU V2 directly as well as several variations where transfer learning is applied in order to adapt the network to different depth sensors while using various data preprocessing and augmentation techniques. We show that a transfer learning approach together with appropriate data processing can enable an accurate modeling of the respective depth sensors.

Depth estimation is one of the basic building blocks for scene understanding. In the case of monocular depth estimation using neural networks, many such approaches are highly hardware dependent because they result in a task- and environment-specific optimizing problem. Most DNN methods use commonly available datasets which leads to overfitting on particular sensor properties. Finding a generalized model with the consideration of different hardware properties of sensors and platforms is challenging if not impossible. For this reason, it is desirable to adapt existing and well-trained models into a new domain in order to let them simulate different depth sensors without the need for large datasets and time-consuming learning. Therefore, a small dataset has been created with the Structure Sensor for evaluating the transferable structural characteristic between neural networks. Finally, two input feature representations for the neural networks are considered to mimic the depth sensor including its artifacts including holes. The results show that a simple domain adaptation technique and a small dataset are adequate to simulate and adapt to a specific domain from a target domain. Therefore, the network is able to accurately predict depth maps as if they were created by a specific depth sensor. This also includes unique artifacts of the sensor, thereby allowing for a plausible simulation of specific depth sensing hardware which is beneficial for areas like prototyping in the context of Augmented Reality.

Augmented reality applications become more and more important to enhance our daily life and workflows. The possibility of showing additional virtual content in a camera stream is helpful for many different use cases like room planning where such applications can offer a simple and intuitive visualization. However, augmented reality applications can suffer from the interference by real objects that may disrupt the user experience. In recent years, there has been research regarding the removal of real objects from camera streams by applying diminished reality techniques. Current approaches are generally limited to flat objects, video streams with little camera movement, or can only remove objects in front of simple and mostly planar backgrounds. In our approach, we show a robust and efficient way to remove a selected 3D object from the camera stream visually. The removal is based on a dense 3D reconstruction of the physical environment stored in a voxel grid that can be created and extended on-the-fly. Hereby, an undesired object can be replaced by a background rendered from the reconstruction allowing for more complex environments than previous approaches. Furthermore, remaining holes by the removal of the object are removed applying an inpainting approach. Finally, we apply color correction to get a seamless transition between the virtual content and the camera image.

Glossy reflections of the surroundings play a major role when trying to achieve a seamless fusion of real and virtual objects in Mixed Reality (MR) environments. Traditionally, the necessary information about the ambiance is captured using mirrored balls, HDR cameras, fish-eye lenses, RGB-D cameras or 360-degree cameras. While these approaches allow for pretty good results, they require a rather complex setup. Our approach is based on a single RGB camera capturing the environmental lighting at a certain location within the scene. Therefore, we apply a precomputation step generating a 360-degree environment map and combine it with a camera-based image stitching for a continuous enhancement and update of the lighting information. We show that our approach allows for realistic and high-quality reflections within an AR/MR environment in real time even on mobile devices.

Real-time capable simultaneous localization and mapping (SLAM) approaches applying consumer hardware have been extensively researched in recent years. Their 3D reconstruction typically applies voxel volumes stored in regular grid hierarchies, sparse voxel octrees or voxel hash tables. They represent the model implicitly in the form of a truncated signed distance function (TSDF). Data integration is usually achieved by stepping through the reconstruction hierarchy from top to bottom and checking voxel grids against the new input data or by rasterizing input data to find associated voxels. For hierarchical representations, a major challenge remains the efficient determination of relevant portions of the reconstruction to be modified by new input data. We present a novel approach efficiently rasterizing input point clouds into intermediate volumes by the GPU. Our technique performs a simple preprocessing step on the input data to properly account for the TSDF representation, allowing for an accurate and hole-free reconstruction. We show that our approach is well suited for a fast integration of new input data into the hierarchical 3D reconstruction, allowing for real-time performance while only slightly increasing memory consumption.

Proper scene inference provides the basis for a seamless integration of virtual objects into the real environment. While widely neglected in many AR/MR environments, previous approaches providing good results were based on rather complex setups, often involving mirrored balls, several HDR cameras, and fish eye lenses to achieve proper light probes. In this paper we present an approach requiring a single RGB-D camera image only for generating glossy reflections on virtual objects. Our approach is based on a partial 3D reconstruction of the real environment combined with a screen-space ray-tracing mechanism. We show that our approach allows for convincing reflections of the real environment as well as mutual reflections between virtual objects of an MR environment.