A wide range of Mixed Reality (MR) applications rely on haptic interaction with real-world objects. These interactions range from grasping and standing on objects, to writing on their surfaces. Accurate alignment of a real-world object with its virtual counterpart enables users to perceive and interact with it both visually and haptically. However, existing registration methods often rely on tracked, external markers, are geometrically constrained, or require complex setups that limit their practical applicability. Hence, we present an integrated pipeline for rapid and accurate registration of arbitrary physical objects in MR. The proposed approach combines custom 3D-printed add-ons for Meta Quest 3 controllers with a point-to-point matching algorithm based on Kabsch optimization, enabling precise selection, calibration, and registration. The method is objectively evaluated using a modern motion capture rig against other state-of-the-art registration techniques. An additional user-centric evaluation helps to understand the general usability of our pipeline. Source code and datasets are released as open resources to encourage reproducibility and further research.

Multi-camera dynamic Augmented Reality (AR) applications require a camera pose estimation to leverage individual information from each camera in one common system. This can be achieved by combining contextual information, such as markers or objects, across multiple views. While commonly cameras are calibrated in an initial step or updated through the constant use of markers, another option is to leverage information already present in the scene, like known objects. Another downside of marker-based tracking is that markers have to be tracked inside the field-of-view (FoV) of the cameras. To overcome these limitations, we propose a constant dynamic camera pose estimation leveraging spatiotemporal FoV overlaps of known objects on the fly. To achieve that, we enhance the state-of-the-art object pose estimator to update our spatiotemporal scene graph, enabling a relation even among non-overlapping FoV cameras. To evaluate our approach, we introduce a multi-camera, multi-object pose estimation dataset with temporal FoV overlap, including static and dynamic cameras. Furthermore, in FoV overlapping scenarios, we outperform the state-of-the-art on the widely used YCB-V and T-LESS dataset in camera pose accuracy. Our performance on both previous and our proposed datasets validates the effectiveness of our marker-less approach for AR applications. The code and dataset are available on https://github.com/roth-hex-lab/IEEE-VR-2026-MultiCam.

A convincing sense of embodiment in virtual reality (VR) is crucial for creating immersive and engaging experiences, as it shapes how users perceive and interact with their virtual bodies. The sense of embodiment is thereby, among others, affected by the shape, appearance, and fidelity of the virtual body. However, achieving convincing avatar appearance remains a challenge for VR applications. One promising solution is Pass-Through Embodiment (PTE), which enables users to see their real bodies in VR. PTE combines depth-based segmentation with the pass-through video stream of video-see-through displays to effectively visualize photon-captured representations of their own bodies. Despite the source-fidelity of the representation, the resolution of integrated depth sensors in Head Mounted Displays (HMD) can produce artifacts at segmentation boundaries, leading to visible aliasing. The perceptual impact of these artifacts on the VR experience remains unexplored. Therefore, in this paper we compare three edge-rendering techniques designed to reduce artifacts without compromising performance. Aside from a soft gradient, we introduce two new methods with a hard and dithered edge. The latter aims to balance the sharpness of hard masks with the smoothness of gradient transitions, without relying on alpha blending. To evaluate those methods in a PTE context, we conducted a within-subjects study that introduces a novel real-mirror paradigm, using an actual physical mirror as reference for reflection. We found significant results in measured presence and embodiment. Subsequent analysis revealed that our dithered cutout approach significantly outperforms hard masks, while no significant difference was found between soft condition. These results suggest a perceptual continuum where dithering and soft blending both effectively reduce visual artifacts through gradient representation. Together with high overall ratings on presence and embodiment across all conditions, these findings confirm PTE as a robust method for supporting embodiment and presence, while highlighting the potential of dithering as a computationally efficient yet perceptually comparable alternative to smooth blending.

Creating a compelling sense of presence and embodiment can enhance the user experience in virtual reality (VR). One method to accomplish this is through self-representation with embodied personalized avatars or video self-avatars. However, these approaches require external hardware and primarily evaluate hand representations in VR across various tasks. We therefore present in this paper an alternative approach: video Pass-Through Embodiment (PTE), which utilizes the per-eye real-time depth map from Head-Mounted Displays (HMDs) traditionally used for Augmented Reality features. This method allows the user's real body to be cut out of the pass-through video stream and be represented in the VR environment without the need for additional hardware. To evaluate our approach, we conducted a between-subjects study involving 40 participants who completed a seated object sorting task using either PTE or a customized avatar. The results show that PTE, despite its limited depth resolution that leads to some visual artifacts, significantly enhances the user's sense of presence and embodiment. In addition, PTE does not negatively affect task performance, cognitive load, or cause VR sickness. These findings imply that video pass-through embodiment offers a practical and efficient alternative to traditional avatar-based methods in VR.

For locomotion in virtual environments (VE) the method of redirected walking (RDW) enables users to explore large virtual areas within a restricted physical space by (almost) natural walking. The trick behind this method is to manipulate the virtual camera in an user-undetectable manner that leads to a change of his movements. If the virtual camera is manipulated too strong then the user recognizes this manipulation and reacts accordingly. We studied the effect of human perception of RDW under the influence of the level of realism in rendering the virtual scene.

In Augmented Reality, interaction with the environment can be achieved with a number of different approaches. In current systems, the most common are hand and gesture inputs. However experimental applications also integrated smartphones as intuitive interaction devices and demonstrated great potential for different tasks. One particular task is constrained object manipulation, for which we conducted a user study. In it we compared standard gesture-based approaches with a touch-based interaction via smartphone. We found that a touch-based interface is significantly more efficient, although gestures are being subjectively more accepted. From these results we draw conclusions on how smartphones can be used to realize modern interfaces in AR.

In this paper, we present SIAM-C, an avatar-mediated communication platform to study socially immersive interaction in virtual environments. The proposed system is capable of tracking, transmitting, representing body motion, facial expressions, and voice via virtual avatars and inherits the transmission of human behaviors that are available in real-life social interactions. Users are immersed using active stereoscopic rendering projected onto a life-size projection plane, utilizing the concept of “fish tank” virtual reality (VR). Our prototype connects two separate rooms and allows for socially immersive avatar-mediated communication in VR.

In this paper, we present a avatar-mediated communication platform for socially immersive interaction in virtual reality (VR). Our approach is based on the combination of body tracking, facial expression tracking and "fishtank" VR. Our prototype enables two remote users to communicate via avatars.