Depth map sequences are commonly compressed using standard video codecs. However, the bitdepth of acquired depth data often exceeds the maximum bitdepth supported by deployed codecs, resulting in significant quantization loss. To overcome this, depth data are frequently mapped on multiple streams before encoding. In this paper, we propose a depth range decomposition of a high-bitdepth source depth map sequence into two lower-bitdepth sequences: a depth band mask sequence and a residual map sequence. The former indicates the depth band for each pixel, while the latter carries the residual within that band. The presented transformation is lossless and exploits the piecewise smoothness of depth data. Both sequences can be compressed independently with any standard video codec, allowing seamless integration into existing video processing pipelines.

Perceptual quality assessment of Dynamic Point Cloud (DPC) contents plays an important role in various Virtual Reality (VR) applications that involve human beings as the end user. Understanding and modeling perceptual quality assessment is greatly enriched by insights from visual attention. However, incorporating aspects of visual attention in DPC quality models is largely unexplored, as ground-truth visual attention data are scarcely available. Besides, testing methods and procedures for collecting visual attention data are still to be agreed on. This article presents a dataset containing subjective opinion scores and visual attention maps of DPCs, collected in a VR environment using eye-tracking technology. Both the quality score and eye-tracking data were collected during a subjective quality assessment experiment, in which subjects were instructed to watch and rate DPCs at various degradation levels under 6 Degrees of Freedom (DoF) inspection, using a head-mounted display. Qualitative interview analysis was also conducted after the experiment. The dataset consists of 50 DPCs, including 5 reference DPCs, with each reference encoded at 3 distortion levels using 3 different codecs (namely G-PCC, V-PCC, CWI-PCL), amounting to a total of 9 degraded version per reference. Additionally, it incorporates 1,000 gaze trials from 40 participants, yielding a total of 15,000 visual attention maps across all the DPCs. We additionally benchmark objective quality metrics originally designed for static point clouds, evaluating their performance in our dataset using two temporal pooling strategies. Furthermore, we employ the visual attention data that are retrieved during our experiment to evaluate whether the performance of widely used objective quality metrics is improved by considering subjective measurements of visual attention. This dataset establishes a link between quality assessment and visual attention within the context of DPC. Moreover, thematic analysis of the interviews helps uncover user behavior and factors impacting perceptual quality for DPC in 6 DoF. This work deepens our understanding of DPC quality assessment and visual attention, driving progress in the realm of VR experiences and perception.

Modern information exchange and telecommunication systems make immersive media increasingly more relevant. Immersive applications introduce challenges in ensuring optimal performance and scalability of media decoding, composition, and synchronization operations. Nowadays, a solution to accommodate media decoding in this context is to use multiple parallel video decoder instances instead of a single video decoder. This would require a robust and predictable decoder management system, coupled with dynamic buffer organization. In practice, however, this is often implemented without any special curation to manage the simultaneous video decoder instances, with resulting behaviour and implications not being fully understood yet. In this paper, we extend our previous work by utilizing the VidBench tool and investigate further the performance of modern chipsets integrated into Android-operated mobile devices in handling multiple video decoder instances running in parallel. Our results indicate the need for suitable synchronization solutions to counter uncertainty and variability across different video decoder instances and devices.

Acquiring depth data has become easily achievable with advancements in depth sensing and depth estimation technologies. As a result, obtaining a depth stream to describe the topology of a corresponding video stream has been considerably simplified. Presence of a depth stream offers numerous benefits, including the integration of advanced visual enhancements to the corresponding video stream in a flexible and efficient manner. This can enrich video-centric applications and facilitate their transition to Augmented Reality (AR) environments, where processing capabilities and battery power are limited. In this paper, we introduce VidDepth, an application developed for mobile devices to demonstrate examples of visual enhancements in video playback scenarios across both traditional and AR settings.

Immersive technologies like eXtended Reality (XR) are the next step in videoconferencing. In this context, understanding the effect of delay on communication is crucial. This article presents the first study on the impact of delay on collaborative tasks using a realistic Social XR system. Specifically, we design an experiment and evaluate the impact of end-to-end delays of 300, 600, 900, 1,200, and 1,500 ms on the execution of a standardized task involving the collaboration of two remote users that meet in a virtual space and construct block-based shapes. To measure the impact of the delay in this communication scenario, objective and subjective data were collected. As objective data, we measured the time required to execute the tasks and computed conversational characteristics by analyzing the recorded audio signals. As subjective data, a questionnaire was prepared and completed by every user to evaluate different factors such as overall quality, perception of delay, annoyance using the system, level of presence, cybersickness, and other subjective factors associated with social interaction. The results show a clear influence of the delay on the perceived quality and a significant negative effect as the delay increases. Specifically, the results indicate that the acceptable threshold for end-to-end delay should not exceed 900 ms. This article additionally provides guidelines for developing standardized XR tasks for assessing interaction in Social XR environments.

Perceptual quality assessment of Dynamic Point Cloud (DPC) contents plays an important role in various Virtual Reality (VR) applications that involve human beings as the end user, understanding and modeling perceptual quality assessment is greatly enriched by insights from visual attention. However, incorporating aspects of visual attention in DPC quality models is largely unexplored, as ground-truth visual attention data is scarcely available. This paper presents a dataset containing subjective opinion scores and visual attention maps of DPCs, collected in a VR environment using eye-tracking technology. The data was collected during a subjective quality assessment experiment, in which subjects were instructed to watch and rate DPCs at various degradation levels under 6 degrees-of-freedom inspection, using a head-mounted display. The dataset comprises 5 reference DPC contents, with each reference encoded at 3 distortion levels using 3 different codecs, amounting to a total of 9 degraded DPC contents. Moreover, it includes 1,000 gaze trials from 40 participants, resulting in 15,000 visual attention maps in total. The curated dataset can serve as authentic benchmark data for assessing the performance of objective DPC quality metrics. Additionally, it establishes a link between quality assessment and visual attention within the context of DPC. This work deepens our understanding of DPC quality and visual attention, driving progress in the realm of VR experiences and perception.

Social virtual reality (VR) allows multiple remote users to interact in a shared space, unveiling new possibilities for communication in immersive environments. Mediascape XR presents a social VR experience that teleports 3D representations of remote users, using volumetric video, to a virtual museum. It enables visitors to interact with cultural heritage artifacts while allowing social interactions in real time between them. The application is designed following a human-centered approach, enabling an interactive, educating, and entertaining experience.

Remote communication has rapidly become a part of everyday life in both professional and personal contexts. However, popular video conferencing applications present limitations in terms of quality of communication, immersion and social meaning. VR remote communication applications offer a greater sense of co-presence and mutual sensing of emotions between remote users. Previous research on these applications has shown that realistic point cloud user reconstructions offer better immersion and communication as compared to synthetic user avatars. However, photorealistic point clouds require a large volume of data per frame and are challenging to transmit over bandwidth-limited networks. Recent research has demonstrated significant improvements to perceived quality by optimizing the usage of bandwidth based on the position and orientation of the user's viewport with user-adaptive streaming. In this work, we developed a real-time VR communication application with an adaptation engine that features tiled user-adaptive streaming based on user behaviour. The application also supports traditional network adaptive streaming. The contribution of this work is to evaluate the impact of tiled user-adaptive streaming on quality of communication, visual quality, system performance and task completion in a functional live VR remote communication system. We performed a subjective evaluation with 33 users to compare the different streaming conditions with a neck exercise training task. As a baseline, we use uncompressed streaming requiring approximately 300 megabits per second and our solution achieves similar visual quality with tiled adaptive streaming at 14 megabits per second. We also demonstrate statistically significant gains in the quality of interaction and improvements to system performance and CPU consumption with tiled adaptive streaming as compared to the more traditional network adaptive streaming.

Technological advances in head-mounted displays and novel real-time 3D acquisition and reconstruction solutions have fostered the development of 6 Degrees of Freedom (6DoF) teleimmersive systems for social VR applications. Point clouds have emerged as a popular format for such applications, owing to their simplicity and versatility; yet, dense point cloud contents are too large to deliver directly over bandwidth-limited networks. In this context, user-adaptive delivery mechanisms are a promising solution to exploit the increased range of motion offered by 6DoF VR applications to yield gains in perceived quality of 3D point cloud user representations, while reducing their bandwidth requirements. In this paper, we perform a user study in VR to quantify the gains adaptive tile selection strategies can bring with respect to non-adaptive solutions. In particular, we define an auxiliary utility function, we employ established methods from the literature and newly-proposed schemes for distributing the bit budget across the tiles, and we evaluate them together with non-adaptive streaming baselines through subjective QoE assessment. Results confirm that considerable gains can be obtained with user-adaptive streaming, achieving bit rate gains of up to 65% with respect to a non-adaptive approach to deliver comparable quality. Our analysis provides useful insights for the design and development of social VR applications.

Point clouds are among popular visual representations for immersive media. However, the vast amount of information generated during their acquisition requires effective compression for practical applications. Although relevant activities from standardization bodies have led to state-of-the-art compression using conventional methods, learning-based encoders have recently emerged as promising solutions with comparable performance while offering additional attractive features. Yet, there is still a large unexplored space for research that can lead to further advances. In this paper, we propose a block prediction module for bit-rate reduction of geometry-only point clouds. Our method exploits spatial redundancies at the decoding stage between block partitions in the point cloud, and predicts a query block using Generative Adversarial Networks. Results show performance improvements of the objective metrics at low bit-rates, after integration in a baseline auto-encoder architecture.

Point cloud is a promising imaging modality for the representation of 3D media. The vast volume of data associated with it requires efficient compression solutions, with lossy algorithms leading to larger bit-rate savings at the expense of visual impairments. While conventional encoding approaches rely on efficient data structures, recent methods have incorporated deep learning for rate-distortion optimization, while inducing perceptual degradations of different natures. To measure the magnitude of such distortions, subjective or objective quality evaluation methodologies are employed. Lately, a remarkable amount of efforts has been devoted to the development of point cloud objective quality metrics, which have been reported to attain high prediction accuracy. However, their performance and generalization capabilities haven’t been evaluated yet in presence of artifacts from learning-based codecs. In this study, we tackle this matter by conducting the first crowdsourcing experiment for point cloud quality reported in the literature, in order to obtain subjective ratings for point cloud models whose topology and color attributes are encoded by both conventional and data-driven methods. Using the subjective scores as ground truth, the performance of a large pool of state-of-the-art quality metrics is rigorously benchmarked, drawing useful insights regarding their efficacy.

In this work we compare two different options to perform on-line subjective quality assessment experiments in the context of the Call for Evidence on JPEG Pleno Point Cloud Coding. A deep-learning based point cloud codec submitted to the Call was tested against current MPEG point cloud compression methods. The first option is based on participants downloading the entire set of stimuli and running a set of scripts in MATLAB to perform the experiment. The second option involves the participants accessing a server on the web and viewing and judging the stimuli using a web browser. Quality scores compiled using both methods were compared showing strong correlation. A second analysis compared the quality scores with those obtained in a prior laboratory-based study using higher resolution screens. The entire study also brought to light each option’s unique advantages and disadvantages that make each one better suited to specific types of subjective evaluation contexts and situations.

Real-time, immersive telecommunication systems are quickly becoming a reality, thanks to the advances in acquisition, transmission, and rendering technologies. Point clouds in particular serve as a promising representation in these type of systems, offering photorealistic rendering capabilities with low complexity. Further development of transmission, coding, and quality evaluation algorithms, though, is currently hindered by the lack of publicly available datasets that represent realistic scenarios of remote communication between people in real-time. In this paper, we release a dynamic point cloud dataset that depicts humans interacting in social XR settings. Using commodity hardware, we capture a total of 45 unique sequences, according to several use cases for social XR. As part of our release, we provide annotated raw material, resulting point cloud sequences, and an auxiliary software toolbox to acquire, process, encode, and visualize data, suitable for real-time applications. The dataset can be accessed via the following link: https://www.dis.cwi.nl/cwipc-sxr-dataset/.

This paper presents a quality evaluation study of point cloud codecs that have been recently standardised by the MPEG committee. In particular, a subjective experiment to assess their performance in terms of bitrate against visual quality is designed and realized in four independent laboratories. The experimental setup of each laboratory varies; yet, the obtained subjective scores exhibit high inter laboratory correlation, confirming that the adopted assessment protocol is robust to equipment selection and viewing conditions, ensuring reliability and facilitating repeatability. Our study confirms the superior compression performance of the MPEG V-PCC, when compared to MPEG G-PCC, in the case of static contents. Finally, results from a benchmark of the most popular objective quality metrics using the obtained subjective scores as ground truth, reveal that the point2plane with mean square error is the most accurate quality predictor, closely followed by the point2point also using mean square error as distance measure.

In this study, we explore the use of virtual reality to subjectively evaluate the visual quality of point cloud contents. To this aim, we develop the PointXR toolbox, a set of Unity applications that can host experiments under variants of interactive and passive evaluation protocols. An auxiliary tool to facilitate the configuration of the supported rendering schemes for point cloud visualization is provided as part of it. Our toolbox is employed to conduct two validating experiments in a virtual environment with 6 degrees of freedom. The purpose is to assess the performance of color encoders that are incorporated in the upcoming MPEG standard on point cloud compression. For this study, we convert a set of mesh models to point cloud contents, and form a high-quality cultural heritage repository, namely, PointXR dataset. A comparison between the adopted protocols and the codecs' performance is carried based on the ratings obtained from both experiments. Finally, interactivity patterns based on behavioral data that were recorded during the evaluations are extracted, and results are discussed. The PointXR toolbox, the PointXR dataset, and the experimental results are made publicly available.

Modelling human visual attention is of great importance in the field of computer vision and has been widely explored for 3D imaging. Yet, in the absence of ground truth data, it is unclear whether such predictions are in alignment with the actual human viewing behavior in virtual reality environments. In this study, we work towards solving this problem by conducting an eye-tracking experiment in an immersive 3D scene that offers 6 degrees of freedom. A wide range of static point cloud models is inspected by human subjects, while their gaze is captured in real-time. The visual attention information is used to extract fixation density maps, that can be further exploited for saliency modelling. To obtain high quality fixation points, we devise a scheme that utilizes every recorded gaze measurement from the two eye-cameras of our set-up. The obtained fixation density maps together with the recorded gaze and head trajectories are made publicly available, to enrich visual saliency datasets for 3D models.

Point clouds are a new modality for representation of plenoptic content and a popular alternative to create immersive media. Despite recent progress in capture, display, storage, delivery and processing, the problem of a reliable approach to subjectively and objectively assess the quality of point clouds is still largely open. In this study, we extend the state of the art in projection-based objective quality assessment of point cloud imaging by investigating the impact of the number of viewpoints employed to assess the visual quality of a content, while discarding information that does not belong to the object under assessment, such as background color. Additionally, we propose assigning weights to the projected views based on interactivity information, obtained during subjective evaluation experiments. In the experiment that was conducted, human observers assessed a carefully selected collection of typical contents, subject to geometry and color degradations due to compression. The point cloud models were rendered using cubes as primitive elements with adaptive sizes based on local neighborhoods. Our results show that employing a larger number of projected views does not necessarily lead to better predictions of visual quality, while user interactivity information can improve the performance.

Recently stakeholders in the area of multimedia representation and transmission have been looking at plenoptic technologies to improve immersive experience. Among these technologies, point clouds denote a volumetric information representation format with important applications in the entertainment, automotive and geographical mapping industries. There is some consensus that state-of-the-art solutions for efficient storage and communication of point clouds are far from satisfactory. This paper describes a study on point cloud quality evaluation, conducted in the context of JPEG Pleno to help define the test conditions of future compression proposals. A heterogeneous set of static point clouds in terms of number of points, geometric structure and represented scenarios were selected and compressed using octree-pruning and a projection-based method, with three different levels of degradation. The models were comprised of both geometrical and color information and were displayed using point sizes large enough to ensure observation of watertight surfaces. The stimuli under assessment were presented to the observers on 2D displays as animations, after defining suitable camera paths to enable visualization of the models in their entirety and realistic consumption. The experiments were carried out in three different laboratories and the subjective scores were used in a series of correlation studies to benchmark objective quality metrics and assess inter-laboratory consistency.

The rise of immersive technologies has been recently fuelled by emerging applications which employ advanced content representations. Among various alternatives, point clouds denote a promising solution which has recently drawn a significant amount of interest, as witnessed by the latest activities of standardization committees. However, subjective and objective quality assessments for this type of content still remain an open problem. In this paper, we introduce a simple yet efficient objective metric to capture perceptual degradations of a distorted point cloud. Correlation with subjective quality assessment scores carried out by human subjects shows the proposed metric to be superior to the state of the art in terms of predicting the visual quality of point clouds under realistic types of distortions, such as octree-based compression.

Recent advances in depth sensing and display technologies, along with the significant growth of interest for augmented and virtual reality applications, lay the foundation for the rapid evolution of applications that provide immersive experiences. In such applications, advanced content representations are required in order to increase the engagement of the user with the displayed imageries. Point clouds have emerged as a promising solution to this aim, due to their efficiency in capturing, storing, delivering and rendering of 3D immersive contents. As in any type of imaging, the evaluation of point clouds in terms of visual quality is essential. In this paper, benchmarking results of the state-of-the-art objective metrics in geometry-only point clouds are reported and analyzed under two different types of geometry degradations, namely Gaussian noise and octree- based compression. Human ratings obtained from two subjective experiments are used as the ground truth. Our results show that most objective quality metrics perform well in the presence of noise, whereas one particular method has high predictive power and outperforms the others after octree-based encoding.

Point clouds are one of the most promising technologies for 3D content representation. In this paper, we describe a study on quality assessment of point clouds, degraded by octree-based compression on different levels. The test contents were displayed using Screened Poisson surface reconstruction, without including any textural information, and they were rated by subjects in a passive way, using a 2D image sequence. Subjective evaluations were performed in five independent laboratories in different countries, with the inter-laboratory correlation analysis showing no statistical differences, despite the different equipment employed. Benchmarking results reveal that the state-of-the-art point cloud objective metrics are not able to accurately predict the expected visual quality of such test contents. Moreover, the subjective scores collected from this experiment were found to be poorly correlated with subjective scores obtained from another test involving visualization of raw point clouds. These results suggest the need for further investigations on adequate point cloud representations and objective Quality assessment tools.

Recently, there has been an increased interest in capture, processing and rendering of visual content in form of point clouds. Among other challenges, subjective and objective quality assessments of point clouds are still open problems. Most proposed subjective quality evaluation methodologies are variants or extensions of counter parts from conventional approaches such as those proposed in various ITU-R and ITU-T recommendations. A key issue with point cloud content is that of rendering and display devices which are thoroughly different from those in other modalities in addition to novel applications which depart from traditional display devices. In this paper, we propose a radically different approach to point cloud subjective quality assessment for point cloud by making use of augmented reality head mounted displays. Beside description of the approach, we show examples of implementation of the proposed methodology and draw conclusions regarding its advantages and drawbacks. Finally, the proposed approach is used in assessing the performance of widely used objective metrics to compute quality of point cloud contents when they undergo various types of distortions such as corruption by noise, simplification and compression.

Point clouds have emerged as a promising solution for immersive representation of 3D contents. In this paper an interactive subjective quality assessment for point clouds is proposed. Quality assessment of geometry information in point clouds subject to realistic types of degradations is performed and correlation between state-of-the-art objective metrics and ground truth subjective scores is investigated. Preliminary results suggest that there is a need for more appropriate objective metrics, since the current solutions are not able to provide accurate predictions of quality for every type of degradations and contents.