Extended Reality (XR) telepresence promises to bring remote instruction closer to the fidelity of in-person guidance by enabling fine-grained visual and spatial cues. However, it remains unclear which representation modality is most effective in supporting hands-on training tasks. This paper presents an exploratory comparative investigation of three such modalities (2D chroma-key compositing, stereoscopic 3D, or RGB-D mesh holograms) within a unified XR telepresence prototype. Through a small-scale study (N=12, 6 pairs of students–teachers) in which a remote instructor guided a trainee through a LEGO assembly task, we evaluated perceived visual quality, overall experience, usability, presence and task time. Our early findings suggest that robust 2D chroma-key compositing currently provides clearer visual guidance than imperfect volumetric reconstruction, while stereoscopic 3D sits between the two. This paper discusses system design considerations, initial empirical insights, and open research challenges for XR remote training. The work is intended as a contribution to ongoing workshop discussions on the evaluation and design of XR telepresence systems.

The lack of skilled workforce also creates new challenges specifically for training and (re)certification of the needed professionals.Extended Reality (XR) technology has emerged as a promising way to improve and enhance education and training.However, with the complexity of technical tasks in manufacturing and other industries, it is not always clear how XR can help in different professional trainings.Therefore, we present our prototype "Ready Learner One" as a means to support both the teaching and quality evaluations of complex technical tasks in a generic way.Our approach utilized 3D stereoscopic capture of the student's and teacher's work surface and presents a fused (blended) view in an Augmented Reality headset (hololens 2) for remote interaction.We utilize a Lego block building task as an example to illustrate our approach.Finally, our prototype will be demonstrated at the ACM IMX 2025 conference as a remote connection between Rio de Janeiro (Brazil) and The Hague (Netherlands).

Video conference applications that allow group communication through video and audio over distance have become commonplace and mainstream. To scale the number of participants in video conferencing systems, the usual practice is to deploy centralized streaming components such as Multipoint Control Units (MCU) or Selective Forwarding Units (SFU). Using these components can become problematic due to significant resource overhead in the server or client. In this paper, we propose a Tile-aware Multipoint Control Unit (T-MCU) that is capable of combining and forwarding streams on a bitstream level. In our solution, multiple VP9 user video streams are combined into a single bitstream, requiring only a single video transmission (single receiving socket, single video decoder, and single rendering texture). Multiple changes (relating to dynamic header and motion vectors) had to be made to the reference VP9 encoder to allow for combining bitstream tiles. These changes are available as open source and are described in this paper, including their impact on encoding performance. Furthermore, all changes are fully compatible with existing VP9 decoders. The paper also presents an experimental design study to evaluate our new T-MCU under different streaming conditions (including the impact on the receiving client) within an immersive 3D communication system utilizing RGBD video data. Our approach significantly reduces the performance requirements of the client (10-20%) and server (~80%) at the cost of increased bandwidth (~16%). Ultimately, the T-MCU allows immersive 3D communication applications to support at least 16+ simultaneous users.

Immersive technologies such as virtual reality have enabled novel forms of education and training, where students can learn new skills in simulated environments. But some specialized training procedures, e.g. ESA-certified soldering, still involve real-world physical processes with physical lab equipment. Such training sessions require students to travel to teaching labs and may interrupt everyday commitments for a longer period of time. There is a desire to make such training procedures more accessible remotely while keeping any student-to-teacher interaction natural, personal, and engaging. This paper presents a prototype for a remote teaching use case by rendering 3D photorealistic representations into the Augmented Reality (AR) glasses of a student. The teacher is captured with a modular RGBD capture application integrated into a web-based immersive communication platform. The integration offers multiple real-time capture calibration and rendering configurations. Our modular platform allows for an easy evaluation of different technical constraints as well as easy testing of the use case itself. Such evaluation may include a direct comparison of different 3D point-cloud and mesh rendering techniques. Additionally, the overall system allows immersive interaction between the student and the teacher, including augmented text messages for non-intrusive notifications. Our platform offers an ideal testbed for both technical and user-centered immersive communication studies.

When operating under adverse conditions or at distant locations, it is not always feasible to obtain the assistance of an expert on site. In such cases, remote expert assistance may provide a solution. Current remote assistance systems employ immersive data visualization or video-based communication. By introducing XR-based multi-user communication system to remote expert assistance solutions, we expect to improve the effectiveness of both the operator and the supporting experts. Our multi-user XR collaboration demo integrates XR and cloud/network technologies. It enables three users to collaborate in a way that makes them feel that they are solving a challenging task together.

Elderly people in care homes face a great lack of contact with their families and loved ones. Social isolation and loneliness are detrimental factors for older people in their health condition, cognitive impairment and quality of life. We previously presented a communication system that facilitated high-quality mediated social contact through an Augmented Reality tool with 3D user capture and rendering on an iPad. We have further enhanced this system and propose a new demonstrator that provides a working communication system between the elderly and family members on a life-size display. A complete end-to-end chain architecture is presented, where capture, transmission, and rendering are enhanced to provide a tool for natural interaction and high social presence. Our system is technically ready and well-adapted to meet the needs of elderly users.

Recent years have seen a new uptake in immersive media and eXtended Reality (XR). And due to a global pandemic, computer-mediated communication over video conferencing tools became a new normal of everyday remote collaboration and virtual meetings. Social XR leverages XR technologies for remote communication and collaboration. But in order for XR to facilitate a high level of (social) presence and thus high-quality mediated social contact between users, we need high-quality 3D representation of users. One approach to providing detailed 3D user representations as new immersive media is to use point clouds or meshes, but these representation formats come with complexity on compression bitrate and processing time. In the example of virtual meetings, compression has to fulfill stringent requirements such as low latency and high quality. As the compression techniques for 3D immersive media steadily advance, it is important to be able to easily compare different compression techniques on their technical and visual merits in an easy way. The proposed demonstrator in this paper is a visualization tool that helps assessing the visual quality of a 3D representation employing various coding schemes. The complete end-to-end rendering/encoding chain can be easily assessed, allowing for subjective testing by showing the differences between the selected encoding parameters. The tool presented in this demo paper offers an improved and easy visual process for the comparison of encoders of immersive media.

Tools and platforms that enable remote communication and collaboration provide a strong contribution to societal challenges. Virtual meetings and conferencing, in particular, can help to reduce commutes and lower our ecological footprint, and can alleviate physical distancing measures in case of global pandemics. In this paper, we outline how to bridge the gap between common video conferencing systems and emerging social VR platforms to allow immersive communication in Virtual Reality (VR). We present a novel VR communication framework that enables remote communication in virtual environments with real-time photorealistic user representation based on colour-and-depth (RGBD) cameras and web browser clients, deployed on common off-the-shelf hardware devices. The paper's main contribution is threefold: (a) a new VR communication framework, (b) a novel approach for real-time depth data transmitting as a 2D grayscale for 3D user representation, including a central MCU-based approach for this new format and (c) a technical evaluation of the system with respect to processing delay, CPU and GPU usage.

Social VR shall allow natural communication between users with high social presence, as if users are in the same room. One way to increase social presence is to add haptic interaction to allow, for example, users to give each other a ”high-five” or to pass documents among them. In this paper, we present our web-based VR communication framework with an added haptic component to simulate touch. The goal of this framework is to enhance the VR communication experience and the social cues exchange between users in VR. We describe our method for rendering haptic feedback within the web-based framework and evaluate the perceived quality of our system with a user survey (with 119 participants). Our proof-of-concept system was rated positively, with the haptic component offering an enhanced quality of the VR experience for 78% of the participants.

Recent developments in key technologies like 5G, Augmented and Virtual Reality (AR/VR) and Tactile Internet result in new possibilities for communication. Particularly, these key digital technologies can enable remote communication and collaboration in remote experiences. In this demo, we work towards 6-degrees of freedom (DoF) photo-realistic shared experiences by introducing a multi-view multi-sensor capture end-to-end system. Our system acts as a baseline end-to-end system for capture, transmission and rendering of volumetric video of user representations. To handle multi-view video processing in a scalable way, we introduce a Multi-point Control Unit (MCU) to shift processing from end devices into the cloud. MCUs are commonly used to bridge videoconferencing connections, and we design and deploy a VR-ready MCU to reduce both upload bandwidth and end-device processing requirements. In our demo, we focus on a remote meeting use case where multiple people can sit around a table to communicate in a shared VR environment.

Virtual Reality (VR) applications are currently suffering several gaps to allow a full breakthrough in the consumer market. One aspect is the social isolation. Many VR experiences do not represent yourself or other users, making it lonely rather than a shared experience. VR applications that do offer a user representation mostly use artificial avatars. These applications may not be well suited for all communication use cases (e.g. at work or with your family). In this paper I present our current work, at TNO and in my PhD research, to create Social VR experiences where both the user and the environment are represented in photo-realistic quality. The aim is not only to allow users to experience VR together, but also to allow new ways of natural communication within VR, with an extended presence and immersion. My PhD research focuses on 3 aspects that are building a Social VR system: capture and processing, transmission and client composition. The main goal of my research is to move processing from the capture and orchestration into the cloud (and particularly into the edge network) to allow Social VR with a large set of users (100+) in one session and to support mobile end devices while maximizing the QoS and QoE.

VR experiences are becoming more social, but many social VR systems represent users as artificial avatars. For use cases such as VR conferencing, photo-realistic representations may be preferred. In this paper, we present ongoing research into social VR experiences with photo-realistic representations of participants and present a web-based social VR framework that extends current video conferencing capabilities with new VR functionalities. We explain the underlying design concepts of our framework and discuss user studies to evaluate the framework in three different scenarios. We show that people are able to use VR communication in real meeting situations and outline our future research to better understand the actual benefits and limitations of our approach, to fully understand the technological gaps that need to be bridged and to better understand the user experience.

In this paper, we describe a user study in which pairs of users watch a video trailer and interact with each other, using two social Virtual Reality (sVR) systems, as well as in a face-to-face condition. The sVR systems are: Facebook Spaces, based on puppet-like customized avatars, and a video-based sVR system using photo-realistic virtual user representations. We collect subjective and objective data to analyze users' Quality of Experience (QoE) and compare their interaction in VR to that observed during the real-life scenario. Our results show that the experience delivered by the video-based sVR system is comparable with real-life settings, while the puppet-based avatars limit the perceived quality of the interaction. Our protocol for QoE assessment is fully documented to allow replication in similar experiments.

Virtual Reality (VR) and 360-degree video are set to become part of the future social environment, enriching and enhancing the way we share experiences and collaborate remotely. While Social VR applications are getting more momentum, most services regarding Social VR focus on animated avatars. In this demo, we present our efforts towards Social VR services based on photo-realistic video recordings. In this demo paper, we focus on two parts, the communication between multiple people (max 3) and the integration of new media formats to represent users as 3D point clouds. We enhance a green screen (chroma key) like cut-out of the person with depth data, allowing point cloud based rendering in the client. Further, the paper presents a user study with 54 people evaluating a three-people communication use case and a technical analysis to move towards 3D representations of users. This demo consists of two shared virtual environments to communicate and interact with others, i.e. i) a 360-degree virtual space with users being represented as 2D video streams (with the background removed) and ii) a 3D space with users being represented as point clouds (based on color and depth video data).

Virtual Reality (VR) and 360-degree video are reshaping the media landscape, creating a fertile business environment. During 2016 new 360-degree cameras and VR headsets entered the consumer market, distribution platforms are being established and new production studios are emerging. VR is evermore becoming a hot topic in research and industry and many new and exciting interactive VR content and experiences are emerging. The biggest gap we see in these experiences are social and shared aspects of VR. In this demo we present our ongoing efforts towards social and shared VR by developing a modular web based VR framework, that extends current video conferencing capabilities with new functionalities of Virtual and Mixed Reality. It allows us to connect two people together for mediated audio-visual interaction, while being able to engage in interactive content. Our framework allows to run extensive technological and user based trials in order to evaluate VR experiences and to build immersive multi-user interaction spaces. Our first results indicate that a high level of engagement and interaction between users is possible in our 360-degree VR set-up utilizing current web technologies.

Social events such as networking parties are excellent opportunities to expand one's social network and make new social ties, as well as simply have fun and enjoy oneself. With the emergence of pervasive technology, we now have the opportunity to discover face-to-face socializing behaviors of individual guests and support them in their socializing goals. Quantitative measures of socializing behaviors will be useful in diverse applications from social therapy to social event management. We propose a system that detects and analyzes group-level socializing behaviors to support real-time and ex post facto social applications deployed in real social event situations. In our approach, we exploit attendees' characteristics of socializing such as proximity and body orientation to determine social proximity among guests and subsequently detect socializing groups. We performed preliminary studies and multiple deployments of our system and example applications in real social networking parties. Evaluation shows the system can detect socializing groups accurately and support applications that satisfy the interests of guests and party hosts.

Video-Mediated group communication is quickly moving from the office to the home, where network conditions might fluctuate. If we are to provide a software component that can, in real-time, monitor the Quality of Experience (QoE), we would have to carry out extensive experiments under different varying (but controllable) conditions. Unfortunately, there are no tools available that provide us the required fined-grained level of control. This paper reports on our efforts implementing such a test bed. The test bed provides the experiment conductor full control over the complete media pipeline, and the possibility of modifying in real-time network and media conditions. Additionally, it has facilities to easily develop an experiment with custom layouts, task integration, and assessment of subjective ratings through questionnaires. We have already used the test bed in a number of evaluations, reported in this paper for discussing the benefits and drawbacks of our solution. The test bed have been proven to be a flexible and effective canvas for better understanding QoE on video-mediated group communication.