The Audiovisual Multimodal Interaction Suite (AMIS) is an open-source dataset and accompanying Unity-based demo implementation designed to aid research on immersive media communication and social XR environments. AMIS features synchronized audiovisual recordings of three actors performing monologues and participating in dyadic conversations across four modalities: talking-head videos, full-body videos, volumetric avatars, and personalized animated avatars. These recordings can be used to simulate scenarios such as traditional video conferences or XR meetings with 3D avatars in controlled and replicable environments. The limitations of existing datasets, which include a restricted number of audiovisual formats, a narrow application focus, and suboptimal inclusion of verbal and non-verbal cues, are addressed by AMIS. With AMIS Studio, a Unity-based demonstrator, researchers can explore the recordings and compare the different audiovisual formats in VR scenes. This paper outlines the creation of AMIS, its design considerations, and how it may be applied in interdisciplinary domains, including cognitive psychology, audiovisual quality assessment, and social XR research.

Group-to-group telepresence systems immerse geographically separated groups in a shared interaction space where remote users are represented as avatars. Notably, such systems allow users to interact with collocated and remote interlocutors simultaneously. In this context, where virtual user representations can be directly compared with real users, we investigate how visual realism (avatar type) and aural realism (presence of spatial audio) affect communication. Furthermore, we examine how communication differs between collocated and remote pairs of interlocutors. In our user study, groups of four participants perform a collaborative conversation task under the aforementioned visual and aural realism conditions. Our results indicate that avatar realism has positive effects on subjective ratings of perceived message understanding and group cohesion, and yields behavioural differences that indicate more interactivity and engagement. Few significant effects of aural realism were observed. Comparisons between collocated and remote communication found that collocated communication was perceived as more effective, but that more visual attention was paid to both remote participants than the collocated user.

Virtual pointing rays are a commonly employed interaction metaphor allowing users of mixed reality applications to select out-of-reach virtual objects. However, fast hand movements can cause the visual representation of the ray to move quickly across the visual field, leading to motion artefacts that cause multiple copies of the ray to appear in discrete locations. In this work, we conduct a user study to investigate whether the conditions for motion artefact occurrence arise during a selection task that participants undertake with a pointing ray. We analyse the recorded gaze and ray movement data from the study to show that pointing actions create the conditions necessary for motion artefacts to appear. In addition, we propose a motion blur effect that we apply to the pointing ray with the aim of mitigating motion artefacts. The blur effect is evaluated in the aforementioned user study and is found to improve the perceived smoothness of ray selection, without affecting selection performance.

RGBD cameras can capture users and their actions in the real world for reconstruction of photo-realistic volumetric avatars that allow rich interaction between spatially distributed telepresence parties in virtual environments. In this paper, we present and evaluate a system design that enables volumetric avatar reconstruction at increased frame rates. We demonstrate that we can overcome the limited capturing frame rate of commodity RGBD cameras such as the Azure Kinect by dividing a set of cameras into two spatio-temporally offset reconstruction groups and implementing a real-time reconstruction pipeline to fuse the temporally offset RGBD image streams. Comparisons of our proposed system against capture configurations possible with the same number of RGBD cameras indicate that it is beneficial to use a combination of spatially and temporally offset RGBD cameras, allowing increased reconstruction frame rates and scene coverage while producing temporally consistent volumetric avatars.

Abstract In this paper we propose a novel and efficient rasterization‐based approach for direct rendering of isosurfaces. Our method exploits the capabilities of task and mesh shader pipelines to identify subvolumes containing potentially visible isosurface geometry, and to efficiently extract primitives which are consumed on the fly by the rasterizer. As a result, our approach requires little preprocessing and negligible additional memory. Direct isosurface rasterization is competitive in terms of rendering performance when compared with ray‐marching‐based approaches, and significantly outperforms them for increasing resolution in most situations. Since our approach is entirely rasterization based, it affords straightforward integration into existing rendering pipelines, while allowing the use of modern graphics hardware features, such as multi‐view stereo for efficient rendering of stereoscopic image pairs for geometry‐bound applications. Direct isosurface rasterization is suitable for applications where isosurface geometry is highly variable, such as interactive analysis scenarios for static and dynamic data sets that require frequent isovalue adjustment.

In this article, we propose a system design and implementation for output-sensitive reconstruction, transmission and rendering of 3D video avatars in distributed virtual environments. In our immersive telepresence system, users are captured by multiple RGBD sensors connected to a server that performs geometry reconstruction based on viewing feedback from remote telepresence parties. This feedback and reconstruction loop enables visibility-aware level-of-detail reconstruction of video avatars regarding geometry and texture data, and considers individual and groups of collocated users. Our evaluation reveals that our approach leads to a significant reduction of reconstruction times, network bandwidth requirements and round-trip times as well as rendering costs in many situations.

We present the collaborative Virtual Reality Neurorobotics Lab, which allows multiple collocated and remote users to experience, discuss and participate in neurorobotic experiments in immersive virtual reality. We describe the coupling of the Neurorobotics Platform of the Human Brain Project with our collaborative virtual reality and 3D telepresence infrastructure and highlight future opportunities arising from our work for research on direct human interaction with simulated robots and brains.