We propose a multi-user Mixed Reality (MR) telepresence system that allows users to interact by seamlessly visualizing remote environments and avatars overlaid onto their local physical space. Building on prior shared-space approaches, our method first aligns overlapping rooms to maximize a shared space –a common area containing matched real and virtual objects where all users can interact. Uniquely, our system extends beyond this shared space by visualizing non-shared spaces, the remaining part of each room, allowing users to inhabit these distinct areas. To address the issue of overlap between non-shared spaces, we dynamically adjust their visibility based on user proximity, using a Voronoi diagram to prioritize subspaces closer to each user. Visualizing the surrounding space of each user conveys spatial context, helping others interpret their behavior within their environment. Visibility is updated in real time as users move, maintaining a coherent sense of spatial awareness. Through a user study, we demonstrate that our system enhances enjoyment, spatial understanding, and presence compared to shared-space-only approaches. Quantitative results further show that our dynamic visibility modulation improves both personal space preservation and space accessibility relative to static methods. Overall, our system provides users with a seamless, dynamically connected, and shared multi-room environment. We provide an system overview and demo video of our work in the supplementary material.

In mixed reality (MR) avatar-mediated telepresence, avatar movement must be adjusted to convey the user's intent in a dissimilar space. This paper presents a novel neural network-based framework designed for translating upper-body gestures, which adjusts virtual avatar movements in dissimilar environments to accurately reflect the user's intended gestures in real-time. Our framework translates a wide range of upper-body gestures, including eye gaze, deictic gestures, free-form gestures, and the transitions between them. A key feature of our framework is its ability to generate natural upper-body gestures for users of different sizes, irrespective of handedness and eye dominance, even though the training is based on data from a single person. Unlike previous methods that require paired motion between users and avatars for training, our framework uses an unpaired approach, significantly reducing training time and allowing for generating a wider variety of motion types. These advantages were made possible by designing two separate networks: the Motion Progression Network, which interprets sparse tracking signals from the user to determine motion progression, and the Upper-body Gesture Network, which autoregressively generates the avatar's pose based on these progressions. We demonstrate the effectiveness of our framework through quantitative comparisons with state-of-the-art methods, qualitative animation results, and a user evaluation in MR telepresence scenarios.

Rapid advances in technology gradually realize immersive mixed-reality (MR) telepresence between distant spaces. This paper presents a novel visual guidance system for avatar-mediated telepresence, directing users to optimal placements that facilitate the clear transfer of gaze and pointing contexts through remote avatars in dissimilar spaces, where the spatial relationship between the remote avatar and the interaction targets may differ from that of the local user. Representing the spatial relationship between the user/avatar and interaction targets with angle-based interaction features, we assign recommendation scores of sampled local placements as their maximum feature similarity with remote placements. These scores are visualized as color-coded 2D sectors to inform the users of better placements for interaction with selected targets. In addition, virtual objects of the remote space are overlapped with the local space for the user to better understand the recommendations. We examine whether the proposed score measure agrees with the actual user perception of the partner's interaction context and find a score threshold for recommendation through user experiments in virtual reality (VR). A subsequent user study in VR investigates the effectiveness and perceptual overload of different combinations of visualizations. Finally, we conduct a user study in an MR telepresence scenario to evaluate the effectiveness of our method in real-world applications.

Avatar-mediated augmented reality telepresence aims to enable distant users to collaborate remotely through avatars. When two spaces involved in telepresence are dissimilar, with different object sizes and arrangements, the avatar movement must be adjusted to convey the user’s intention rather than directly following their motion, which poses a significant challenge. In this paper, we propose a novel neural network-based framework for real-time retargeting of users’ deictic motions (pointing at and touching objects) to virtual avatars in dissimilar environments. Our framework translates the user’s deictic motion, acquired from a sparse set of tracking signals, to the virtual avatar’s deictic motion for a corresponding remote object in real-time. One of the main features of our framework is that a single trained network can generate natural deictic motions for various sizes of users. To this end, our network includes two sub-networks: AngleNet and MotionNet. AngleNet maps the angular state of the user’s motion into a latent representation, which is subsequently converted by MotionNet into the avatar’s pose, considering the user’s scale. We validate the effectiveness of our method in terms of deictic intention preservation and movement naturalness through quantitative comparison with alternative approaches. Additionally, we demonstrate the utility of our approach through several AR telepresence scenarios.