A wide range of Mixed Reality (MR) applications rely on haptic interaction with real-world objects. These interactions range from grasping and standing on objects, to writing on their surfaces. Accurate alignment of a real-world object with its virtual counterpart enables users to perceive and interact with it both visually and haptically. However, existing registration methods often rely on tracked, external markers, are geometrically constrained, or require complex setups that limit their practical applicability. Hence, we present an integrated pipeline for rapid and accurate registration of arbitrary physical objects in MR. The proposed approach combines custom 3D-printed add-ons for Meta Quest 3 controllers with a point-to-point matching algorithm based on Kabsch optimization, enabling precise selection, calibration, and registration. The method is objectively evaluated using a modern motion capture rig against other state-of-the-art registration techniques. An additional user-centric evaluation helps to understand the general usability of our pipeline. Source code and datasets are released as open resources to encourage reproducibility and further research.

A convincing sense of embodiment in virtual reality (VR) is crucial for creating immersive and engaging experiences, as it shapes how users perceive and interact with their virtual bodies. The sense of embodiment is thereby, among others, affected by the shape, appearance, and fidelity of the virtual body. However, achieving convincing avatar appearance remains a challenge for VR applications. One promising solution is Pass-Through Embodiment (PTE), which enables users to see their real bodies in VR. PTE combines depth-based segmentation with the pass-through video stream of video-see-through displays to effectively visualize photon-captured representations of their own bodies. Despite the source-fidelity of the representation, the resolution of integrated depth sensors in Head Mounted Displays (HMD) can produce artifacts at segmentation boundaries, leading to visible aliasing. The perceptual impact of these artifacts on the VR experience remains unexplored. Therefore, in this paper we compare three edge-rendering techniques designed to reduce artifacts without compromising performance. Aside from a soft gradient, we introduce two new methods with a hard and dithered edge. The latter aims to balance the sharpness of hard masks with the smoothness of gradient transitions, without relying on alpha blending. To evaluate those methods in a PTE context, we conducted a within-subjects study that introduces a novel real-mirror paradigm, using an actual physical mirror as reference for reflection. We found significant results in measured presence and embodiment. Subsequent analysis revealed that our dithered cutout approach significantly outperforms hard masks, while no significant difference was found between soft condition. These results suggest a perceptual continuum where dithering and soft blending both effectively reduce visual artifacts through gradient representation. Together with high overall ratings on presence and embodiment across all conditions, these findings confirm PTE as a robust method for supporting embodiment and presence, while highlighting the potential of dithering as a computationally efficient yet perceptually comparable alternative to smooth blending.

Creating a compelling sense of presence and embodiment can enhance the user experience in virtual reality (VR). One method to accomplish this is through self-representation with embodied personalized avatars or video self-avatars. However, these approaches require external hardware and primarily evaluate hand representations in VR across various tasks. We therefore present in this paper an alternative approach: video Pass-Through Embodiment (PTE), which utilizes the per-eye real-time depth map from Head-Mounted Displays (HMDs) traditionally used for Augmented Reality features. This method allows the user's real body to be cut out of the pass-through video stream and be represented in the VR environment without the need for additional hardware. To evaluate our approach, we conducted a between-subjects study involving 40 participants who completed a seated object sorting task using either PTE or a customized avatar. The results show that PTE, despite its limited depth resolution that leads to some visual artifacts, significantly enhances the user's sense of presence and embodiment. In addition, PTE does not negatively affect task performance, cognitive load, or cause VR sickness. These findings imply that video pass-through embodiment offers a practical and efficient alternative to traditional avatar-based methods in VR.

Wave-optical phenomena, such as diffraction, significantly impact the visual appearance of surfaces. Despite their importance, wave-optical reflection models are rare and computationally expensive. Recently, we presented a real-time model that accounts for diffraction-induced color shifts and speckle. Given that diffraction phenomena are highly dependent on illumination and viewing directions, as well as stereoscopic vision, we developed a VR demo to evaluate the new model. This demo shows the substantial impact of diffraction on the appearance of rough surfaces, particularly in stereoscopic viewing.

Virtual Reality (VR) systems rely on established real-time rendering techniques to uphold a strict performance budget of a few milliseconds per frame. The utilized techniques are based on approximations that trade speed-up for the correctness of the generated depth cues. The visual impact of these trade-offs, however, remains largely unexplored in the context of VR. In this paper we focus specifically on the perception of distorted specular reflections. Our research goal is to quantify the impairment of common reflection approximations on the resulting visual quality in VR. To this end, a subjective quality assessment is conducted, where participants rate the visual difference and visual annoyance of multiple established reflection approximations in a side-by-side comparison to a raytraced reference solution. We introduce the first dataset of its kind, containing 9 specular reflection approximations evaluated for two object types and three material smoothness values. Our results show the quality benefits associated with currently used reflection methods, and we discuss the implications for rendering VR environments.

Abstract Simulating light–matter interaction is a fundamental problem in computer graphics. A particular challenge is the simulation of light interaction with rough surfaces due to diffraction and multiple scattering phenomena. To properly model these phenomena, wave‐optics have to be considered. Nevertheless, the most accurate BRDF models, including wave‐optics, are computationally expensive, and the resulting renderings have not been systematically compared to real‐world measurements. This work sheds more light on reflectance variations due to surface roughness. More specifically, we look at wavelength shifts that lead to reddish and blueish appearances. These wavelength shifts have been scarcely reported in the literature, and, in this paper, we provide the first thorough analysis from precise measured data. We measured the spectral in‐plane BRDF of aluminium samples with varying roughness and further acquired the surface topography with a confocal microscope. The measurements show that the rough samples have, on average, a reddish and blueish appearance in the forward and back‐scattering, respectively. Our investigations conclude that this is a diffraction‐based effect that dominates the overall appearance of the samples. Simulations using a virtual gonioreflectometer further confirm our claims. We propose a linear model that can closely fit such phenomena, where the slope of the wavelength shifts depends on the incident and reflection direction. Based on these insights, we developed a simple BRDF model based on the Cook–Torrance model that considers such wavelength shifts.

This paper presents a system for a simulated billiard game with two players and an emphasis on haptic feedback. We devised a feedback box that is responsible for generating the inputs and providing immediate haptic feedback to the user. The simulation runs as an AR application and the player can use a real queue to hit the real ball. Although the haptic feedback is precise due to the usage of a real billiard ball and queue, the input accuracy of the angle and impulse measurement is limited.

This paper presents an image-based rendering approach to accelerate rendering time of virtual scenes containing a large number of complex high poly count objects. Our approach replaces complex objects by impostors, light-weight image-based representations leveraging geometry and shading related processing costs. In contrast to their classical implementation, our impostors are specifically designed to work in Virtual-, Augmented- and Mixed Reality scenarios (XR for short), as they support stereoscopic rendering to provide correct depth perception. Motion parallax of typical head movements is compensated by using a ray marched parallax correction step. Our approach provides a dynamic run-time recreation of impostors as necessary for larger changes in view position. The dynamic run-time recreation is decoupled from the actual rendering process. Hence, its associated processing cost is therefore distributed over multiple frames. This avoids any unwanted frame drops or latency spikes even for impostors of objects with complex geometry and many polygons. In addition to the significant performance benefit, our impostors compare favorably against the original mesh representation, as geometric and textural temporal aliasing artifacts are heavily suppressed.

Stereoscopic rendering is a prominent feature of virtual reality applications to generate depth cues and to provide depth perception in the virtual world. However, straight-forward stereo rendering methods usually are expensive since they render the scene from two eye-points which in general doubles the frame times. This is particularly problematic since virtual reality sets high requirements for real-time capabilities and image resolution. Hence, this paper presents a hybrid rendering system that combines classic rasterization and real-time ray-tracing to accelerate stereoscopic rendering. The system reprojects the pre-rendered left half of the stereo image pair into the right perspective using a forward grid warping technique and identifies resulting reprojection errors, which are then efficiently resolved by adaptive real-time ray-tracing. A final analysis shows that the system achieves a significant performance gain, has a negligible quality impact, and is suitable even for higher rendering resolutions.

This paper investigates the effects of normal mapping on the perception of geometric depth between stereoscopic and non-stereoscopic views. Results show, that in a head-tracked environment, the addition of binocular disparity has no impact on the error rate in the detection of normal-mapped geometry. It does however significantly shorten the detection time.

We propose the tool IslandViz for exploring modular software systems in virtual reality. We use an island metaphor, which represents every module as a distinct island. The resulting island system is displayed in the confines of a virtual table, where users can explore the software visualization on multiple levels of granularity by performing navigational tasks. Our approach allows users to get a first overview about the complexity of an OSGi-based software system by interactively exploring its modules as well as the dependencies between them.

We present an approach for exploring OSGi-based software systems in virtual reality. We employ an island metaphor, which represents every module as a distinct island. The resulting island system is displayed in the confines of a virtual table, where users can explore the software visualization on multiple levels of granularity by performing intuitive navigational tasks. Our approach allows users to get a first overview about the complexity of an OSGi-based software system by interactively exploring its modules as well as the dependencies between them.

For locomotion in virtual environments (VE) the method of redirected walking (RDW) enables users to explore large virtual areas within a restricted physical space by (almost) natural walking. The trick behind this method is to manipulate the virtual camera in an user-undetectable manner that leads to a change of his movements. If the virtual camera is manipulated too strong then the user recognizes this manipulation and reacts accordingly. We studied the effect of human perception of RDW under the influence of the level of realism in rendering the virtual scene.

In Augmented Reality, interaction with the environment can be achieved with a number of different approaches. In current systems, the most common are hand and gesture inputs. However experimental applications also integrated smartphones as intuitive interaction devices and demonstrated great potential for different tasks. One particular task is constrained object manipulation, for which we conducted a user study. In it we compared standard gesture-based approaches with a touch-based interaction via smartphone. We found that a touch-based interface is significantly more efficient, although gestures are being subjectively more accepted. From these results we draw conclusions on how smartphones can be used to realize modern interfaces in AR.

Abstract Physically based rendering uses principles of physics to model the interaction of light with matter. Even though it is possible to achieve photorealistic renderings, it often fails to be predictive. There are two major issues: first, there is no analytic material model that considers all appearance critical characteristics; second, light is in many cases described by only 3 RGB‐samples. This leads to the problem that there are different models for different material types and that wavelength dependent phenomena are only approximated. In order to be able to analyze the influence of both problems on the appearance of real world materials, an accurate comparison between rendering and reality is necessary. Therefore, in this work, we acquired a set of precisely and spectrally resolved ground truth data. It consists of the precise description of a new developed reference scene including isotropic BRDFs of 24 color patches, as well as the reference measurements of all patches under 13 different angles inside the reference scene. Our reference data covers rough materials with many different spectral distributions and various illumination situations, from direct light to indirect light dominated situations.

In this paper, we present SIAM-C, an avatar-mediated communication platform to study socially immersive interaction in virtual environments. The proposed system is capable of tracking, transmitting, representing body motion, facial expressions, and voice via virtual avatars and inherits the transmission of human behaviors that are available in real-life social interactions. Users are immersed using active stereoscopic rendering projected onto a life-size projection plane, utilizing the concept of “fish tank” virtual reality (VR). Our prototype connects two separate rooms and allows for socially immersive avatar-mediated communication in VR.

In this paper, we compare a full body marker set with a reduced rigid body marker set supported by inverse kinematics. We measured system latency, illusion of virtual body ownership, and task load in an applied scenario for inducing acrophobia. While not showing a significant change in body ownership or task performance, results do show that latency and task load are reduced when using the rigid body inverse kinematics solution. The approach therefore has the potential to improve virtual reality experiences.

In this paper, we describe an experimental method to investigate the effects of reduced social information and behavioral channels in immersive virtual environments with full-body avatar embodiment. We compared physical-based and verbal-based social interactions in real world (RW) and virtual reality (VR). Participants were represented by abstract avatars that did not display gaze, facial expressions or social cues from appearance. Our results show significant differences in terms of presence and physical performance. However, differences in effectiveness in the verbal task were not present. Participants appear to efficiently compensate for missing social and behavioral cues by shifting their attentions to other behavioral channels.

In this paper, we present a avatar-mediated communication platform for socially immersive interaction in virtual reality (VR). Our approach is based on the combination of body tracking, facial expression tracking and "fishtank" VR. Our prototype enables two remote users to communicate via avatars.

Collision detection is an enabling technology for many virtual environments, games, and virtual prototyping applications containing some kind of physically-based simulation (such as rigid bodies, cloth, surgery, etc.). The persistent interest in collision detection by both academia and industry is due to the large variety of applications as well as new challenges in new application domains, such as virtual clothes fitting. This tutorial will give an overview of the different classes of algorithms, and then provide attendees with in-depth knowledge of some of the important algorithms within each class, focusing in particular on recent approaches, such as stochastic methods or image-space techniques. Attendees will learn to select the one most appropriate for their application, and implement it with only very little further research. The intended audience are practitioners and students working in 3D computer graphics. Topics to be addressed - hierarchical collision detection - stochastic methods - distance fields - spatial subdivisions - image-space techniques - continuous collision detection - non-polygonal object representations (e.g., point clouds)

Abstract Interactive environments for dynamically deforming objects play an important role in surgery simulation and entertainment technology. These environments require fast deformable models and very efficient collision handling techniques. While collision detection for rigid bodies is well investigated, collision detection for deformable objects introduces additional challenging problems. This paper focuses on these aspects and summarizes recent research in the area of deformable collision detection. Various approaches based on bounding volume hierarchies, distance fields and spatial partitioning are discussed. In addition, image‐space techniques and stochastic methods are considered. Applications in cloth modeling and surgical simulation are presented.