Tools and platforms that enable remote communication and collaboration provide a strong contribution to societal challenges. Virtual meetings and conferencing, in particular, can help to reduce commutes and lower our ecological footprint, and can alleviate physical distancing measures in case of global pandemics. In this paper, we outline how to bridge the gap between common video conferencing systems and emerging social VR platforms to allow immersive communication in Virtual Reality (VR). We present a novel VR communication framework that enables remote communication in virtual environments with real-time photorealistic user representation based on colour-and-depth (RGBD) cameras and web browser clients, deployed on common off-the-shelf hardware devices. The paper's main contribution is threefold: (a) a new VR communication framework, (b) a novel approach for real-time depth data transmitting as a 2D grayscale for 3D user representation, including a central MCU-based approach for this new format and (c) a technical evaluation of the system with respect to processing delay, CPU and GPU usage.

During the COVID-19 pandemic, many nursing homes had to restrict visitations. This had a major negative impact on the wellbeing of residents and their family members. In response, residents and family members increasingly resorted to mediated communication to maintain social contact. To facilitate high-quality mediated social contact between residents in nursing homes and remote family members, we developed an augmented reality (AR)-based communication tool. In this study, we compared the user experience (UX) of AR-communication with that of video calling, for 10 pairs of residents and family members. We measured enjoyment, spatial presence and social presence, attitudes, behavior and conversation duration. In the AR-communication condition, residents perceived a 3D projection of their remote family member onto a chair placed in front of them. In the video calling condition, the family member was shown using 2D video. In both conditions, the family member perceived the resident in the video calling mode on a 2D screen. While residents reported no differences in their UX between both conditions, family members reported higher spatial presence for the AR-communication condition compared to video-calling. Conversation durations were significantly longer during AR-communication than during video calling. We tentatively suggest that there may be (unconscious) differences in UX during AR-based communication compared to video calling.

VR experiences are becoming more social, but many social VR systems represent users as artificial avatars. For use cases such as VR conferencing, photo-realistic representations may be preferred. In this paper, we present ongoing research into social VR experiences with photo-realistic representations of participants and present a web-based social VR framework that extends current video conferencing capabilities with new VR functionalities. We explain the underlying design concepts of our framework and discuss user studies to evaluate the framework in three different scenarios. We show that people are able to use VR communication in real meeting situations and outline our future research to better understand the actual benefits and limitations of our approach, to fully understand the technological gaps that need to be bridged and to better understand the user experience.

Virtual Reality (VR) and 360-degree video are set to become part of the future social environment, enriching and enhancing the way we share experiences and collaborate remotely. While Social VR applications are getting more momentum, most services regarding Social VR focus on animated avatars. In this demo, we present our efforts towards Social VR services based on photo-realistic video recordings. In this demo paper, we focus on two parts, the communication between multiple people (max 3) and the integration of new media formats to represent users as 3D point clouds. We enhance a green screen (chroma key) like cut-out of the person with depth data, allowing point cloud based rendering in the client. Further, the paper presents a user study with 54 people evaluating a three-people communication use case and a technical analysis to move towards 3D representations of users. This demo consists of two shared virtual environments to communicate and interact with others, i.e. i) a 360-degree virtual space with users being represented as 2D video streams (with the background removed) and ii) a 3D space with users being represented as point clouds (based on color and depth video data).

Virtual Reality (VR) and 360-degree video are reshaping the media landscape, creating a fertile business environment. During 2016 new 360-degree cameras and VR headsets entered the consumer market, distribution platforms are being established and new production studios are emerging. VR is evermore becoming a hot topic in research and industry and many new and exciting interactive VR content and experiences are emerging. The biggest gap we see in these experiences are social and shared aspects of VR. In this demo we present our ongoing efforts towards social and shared VR by developing a modular web based VR framework, that extends current video conferencing capabilities with new functionalities of Virtual and Mixed Reality. It allows us to connect two people together for mediated audio-visual interaction, while being able to engage in interactive content. Our framework allows to run extensive technological and user based trials in order to evaluate VR experiences and to build immersive multi-user interaction spaces. Our first results indicate that a high level of engagement and interaction between users is possible in our 360-degree VR set-up utilizing current web technologies.

The growing popularity and increasing performance of mobile devices is transforming the way in which media can be consumed, from single device playback to orchestrated multi-stream experiences across multiple devices. One of the biggest challenges in realizing such immersive media experience is the dynamic management of synchronicity between associated media streams. This is further complicated by the faceted aspects of user perception and heterogeneity of user devices and networks. This paper introduces a QoE-aware open inter-stream media synchronization framework (IMSync). IMSync employs efficient monitoring and control mechanisms, as well as a bespoke QoE impact model derived from subjective user experiments. Given a current lag, IMSync's aim is to use the impact model to determine a good catch-up strategy that minimizes detrimental impact on QoE. The impact model balances the accumulative impact of re-synchronization processes and the degree of non-synchronicity to ensure the QoE. Experimental results verify the run-time performance of the framework as a foundation for immersive media experience in open N-Screens cloud.

Inter-Destination Multimedia Synchronization(IDMS) is a key requirement to enable satisfying group shared media experiences. This paper presents the design and simulation of a Distributed Control Scheme (DCS) for IDMS. In this scheme, geographically dispersed receivers exchange(multicast) RTCP reports on their arrival and play out timing for RTP packets, and each one of them uses the collected reports from the other receivers belonging to the same logical group (or cluster) to adaptively adjust its local play out timing such that an overall synchronization status (within allowable limits) is maintained. Also, the strengths (interactivity, flexibility, scalability, and robustness) and weaknesses (traffic overhead, security) of a DCS for IDMS are discussed, in comparison with centralized approaches. Moreover, a simple technique to enhance the performance in terms of coherence when using the DCS for IDMS is proposed. Simulation results prove the effectiveness of the proposed control scheme and techniques to guarantee concurrently synchronized play outpoints for independent clusters of receivers.

Inter-Destination Media Synchronization (IDMS) is essential in the emerging media consumption paradigm, which is radically evolving from passive and isolated services towards dynamic and interactive group shared experiences. This paper concentrates on improving a standardized RTP/RTCP-based solution for IDMS. In particular, novel Early Event-Driven (EED) RTCP feedback reporting mechanisms are designed to overcome latency issues and to enable higher flexibility, dynamism and accuracy when using RTP/RTCP for IDMS. The faster reaction on dynamic situations (e.g., detection of asynchrony or channel change delays) and a finer granularity for synchronizing media-related events, while preserving the RTCP bandwidth bounds, are validated through simulation tests.