When operating under adverse conditions or at distant locations, it is not always feasible to obtain the assistance of an expert on site. In such cases, remote expert assistance may provide a solution. Current remote assistance systems employ immersive data visualization or video-based communication. By introducing XR-based multi-user communication system to remote expert assistance solutions, we expect to improve the effectiveness of both the operator and the supporting experts. Our multi-user XR collaboration demo integrates XR and cloud/network technologies. It enables three users to collaborate in a way that makes them feel that they are solving a challenging task together.

Advances in robotics and multisensory displays allow extending telepresence ambitions beyond only the feeling of being present at a remote location In this paper, we discuss what may lie beyond telepresence and how we can transport both the functional and social self of a user. We introduce the embodiment illusion and its potential contribution to task performance and list important cues to evoke this illusion, including synchronicity in multisensory information, a first-person visual perspective, and a human-like visual appearance and anatomy of the telepresence robot. We also introduce the concept of social presence and the important bidirectional social cues it needs, including eye contact, facial expression, posture, gestures, and social touch. For all these multisensory and social cues, we explain how they can be implemented in a telepresence system and describe our solution consisting of a closed control pod and a humanoid telepresence robot.

Virtual reality (VR) has created a new and rich medium for people to meet each other digitally. In VR, people can choose from a broad range of representations. In several cases, it is important to provide users with avatars that are a lifelike representation of themselves, to increase the user experience and effectiveness of communication. In this work, we propose a pipeline for generating a realistic and expressive avatar from a single reference image. The pipeline consists of a blendshape-based avatar combined with two deep learning improvements. The first improvement module runs offline and improves the texture map of the base avatar. The second module runs inference in real-time at the rendering stage and performs a style transfer to the avatar’s eyes. The deep learning modules effectively improve the visual representation of the avatar and show how AI techniques can be integrated with traditional animation methods to generate realistic human avatars for social VR.

This paper presents a proof-of-concept of a robotic teleoperation system, that provides the human operator a thermal sense in addition to the visual sense. With a sensor suite comprising a stereo camera, 360° camera and long-wave infra-red camera, our demonstrator pushes the boundaries of virtual-reality situational awareness by bringing not only 3D visual content but also a 360° thermal experience to the operator. The visual channel of our robotic teleoperation system is represented through a head-mounted-display and the thermal channel is displayed through directional heaters in the operator cockpit and a thermal glove. Initial tests showed that an operator successfully experienced a 360° remote environment, correctly distinguished between and interacted with hot and cold objects, and could notice the presence of nearby people outside her direct field-of-view, based on their emitted heat.

Recent years have seen a new uptake in immersive media and eXtended Reality (XR). And due to a global pandemic, computer-mediated communication over video conferencing tools became a new normal of everyday remote collaboration and virtual meetings. Social XR leverages XR technologies for remote communication and collaboration. But in order for XR to facilitate a high level of (social) presence and thus high-quality mediated social contact between users, we need high-quality 3D representation of users. One approach to providing detailed 3D user representations as new immersive media is to use point clouds or meshes, but these representation formats come with complexity on compression bitrate and processing time. In the example of virtual meetings, compression has to fulfill stringent requirements such as low latency and high quality. As the compression techniques for 3D immersive media steadily advance, it is important to be able to easily compare different compression techniques on their technical and visual merits in an easy way. The proposed demonstrator in this paper is a visualization tool that helps assessing the visual quality of a 3D representation employing various coding schemes. The complete end-to-end rendering/encoding chain can be easily assessed, allowing for subjective testing by showing the differences between the selected encoding parameters. The tool presented in this demo paper offers an improved and easy visual process for the comparison of encoders of immersive media.

Tools and platforms that enable remote communication and collaboration provide a strong contribution to societal challenges. Virtual meetings and conferencing, in particular, can help to reduce commutes and lower our ecological footprint, and can alleviate physical distancing measures in case of global pandemics. In this paper, we outline how to bridge the gap between common video conferencing systems and emerging social VR platforms to allow immersive communication in Virtual Reality (VR). We present a novel VR communication framework that enables remote communication in virtual environments with real-time photorealistic user representation based on colour-and-depth (RGBD) cameras and web browser clients, deployed on common off-the-shelf hardware devices. The paper's main contribution is threefold: (a) a new VR communication framework, (b) a novel approach for real-time depth data transmitting as a 2D grayscale for 3D user representation, including a central MCU-based approach for this new format and (c) a technical evaluation of the system with respect to processing delay, CPU and GPU usage.

Social VR shall allow natural communication between users with high social presence, as if users are in the same room. One way to increase social presence is to add haptic interaction to allow, for example, users to give each other a ”high-five” or to pass documents among them. In this paper, we present our web-based VR communication framework with an added haptic component to simulate touch. The goal of this framework is to enhance the VR communication experience and the social cues exchange between users in VR. We describe our method for rendering haptic feedback within the web-based framework and evaluate the perceived quality of our system with a user survey (with 119 participants). Our proof-of-concept system was rated positively, with the haptic component offering an enhanced quality of the VR experience for 78% of the participants.

Recent developments in key technologies like 5G, Augmented and Virtual Reality (AR/VR) and Tactile Internet result in new possibilities for communication. Particularly, these key digital technologies can enable remote communication and collaboration in remote experiences. In this demo, we work towards 6-degrees of freedom (DoF) photo-realistic shared experiences by introducing a multi-view multi-sensor capture end-to-end system. Our system acts as a baseline end-to-end system for capture, transmission and rendering of volumetric video of user representations. To handle multi-view video processing in a scalable way, we introduce a Multi-point Control Unit (MCU) to shift processing from end devices into the cloud. MCUs are commonly used to bridge videoconferencing connections, and we design and deploy a VR-ready MCU to reduce both upload bandwidth and end-device processing requirements. In our demo, we focus on a remote meeting use case where multiple people can sit around a table to communicate in a shared VR environment.

3D visual understanding of the surrounding environment is vital for successful mobile robotic tasks such as autonomous navigation or general object interaction. However, current systems have limited perceptual capabilities in the sense that they are not very well adaptable to unknown environments. Human operators, on the other hand, are experts in adapting to previously unknown information. Hence, human-robot teaming in which the human helps the robot to adapt to new environments and the robot assists in automated object recognition to efficiently feed the control environment of the operator is advantageous. In this work, we propose an object recognition and localization system for mobile robots, based on deep learning, and we study the adaptation of the resulting robotic perception to a new environment. We propose two methods to teach the robot a new object category: using prior knowledge and using limited operator input. We conducted several experiments to show the feasibility of proposed methods.

Virtual Reality (VR) and 360-degree video are set to become part of the future social environment, enriching and enhancing the way we share experiences and collaborate remotely. While Social VR applications are getting more momentum, most services regarding Social VR focus on animated avatars. In this demo, we present our efforts towards Social VR services based on photo-realistic video recordings. In this demo paper, we focus on two parts, the communication between multiple people (max 3) and the integration of new media formats to represent users as 3D point clouds. We enhance a green screen (chroma key) like cut-out of the person with depth data, allowing point cloud based rendering in the client. Further, the paper presents a user study with 54 people evaluating a three-people communication use case and a technical analysis to move towards 3D representations of users. This demo consists of two shared virtual environments to communicate and interact with others, i.e. i) a 360-degree virtual space with users being represented as 2D video streams (with the background removed) and ii) a 3D space with users being represented as point clouds (based on color and depth video data).

Three-Dimensional face recognition is a challenging task with a large number of proposed solutions [1, 2]. With variations in pose and expression the identification of a face scan based on 3D geome-try is difficult. To improve on this task and to evaluate existing face

Many 3D face matching techniques have been developed to perform face recognition. Among these techniques are variants of 3D facial curve matching, which are techniques that reduce the amount of face data to one or a few 3D curves. The face’s central profile, for instance, proved to work well. However, the selection of the optimal set of 3D curves and the best way to match them is still under-exposed. We propose a 3D face matching framework that allows profile and contour based face matching. Using this framework we evaluate profile and contour types including those described in literature, and select subsets of facial curves for effective and efficient face matching. Results on the 3D face retrieval track of SHREC’07 (the 3D SHape Retrieval Contest) shows the highest mean average precision achieved so far, using only three facial curves of 45 samples each.

In this work we compute the similarity of 3D faces using a set of eight contour curves. These contours were selected and matched using our 3D face matching framework. In previous work, we performed extensive research to the selection of distinctive facial curves for 3D face matching. To relate the performance of several of these curves to other face matching methods, we participated the Shape Retrieval Contest (SHREC) of 3D face scans. Within this contest we have used a set of eight C-contours and tested their face retrieval performance using two different distance measures. In an attempt to increase the expression invariance of these curves, we employed our 3D face matching framework to match either 100% of the selected features or the subset of the best 60% of the selected features. Results show that the selected distance measure can have a great influence on the distinctiveness of facial curves. In case of large variations in facia) expressiveness, the subset of the best 60% of the features increases the overall performance. With a recognition rate of 91.1% and a mean average precision of 0.693 our method performs reasonably well compared to other methods.

Since 2006 the 3D shape retrieval contest SHREC has been organized. The general objective is to evaluate the effectiveness of 3D shape retrieval algorithms. 3D media retrieval is overlooked in most commercial search engines, while at the same time it is expected to represent a huge amount of traffic and data stored in the Internet. Recent advances in technology have made available cost-effective scanning devices that could not even be imagined a decade ago. It is now possible to acquire 3D data of a physical object in a few seconds and produce a digital model of its geometry that can be easily shared on the Internet. On the other hand, most PCs connected to the Internet are nowadays equipped with high-performance 3D graphics hardware, that support rendering, interaction and processing capabilities from home environments to enterprise scenarios.

This paper presents a new multiview alignment algorithm that performs both the coarse and fine alignment of unordered sets of range scans. Our algorithm selects quadruples of range scans, which have feature points of their 2D projections in common. These quadruples are then verified using an Iterative Closest Point (ICP) algorithm. The accepted quadruples form incomplete models of an object that can be aligned using isometries of the Principal Component Analysis (PCA) in combination with an ICP algorithm. The output of our method is a set of finely aligned meshes. Our method was applied to range scans of different clusters of objects varying in the type of acquisition system, the number of range scans, the scan resolution, and scan accuracy. Results show that our method is both effective and efficient for the alignment of meshes: it is capable of aligning object meshes with various properties, aligns the majority of meshes without a priori knowledge, and doesn't require a multiview ICP algorithm to improve the final alignment.