Point clouds are a fundamental format for immersive experiences, posing significant challenges for storage and transmission. Unlike 2D image compression, 3D point clouds are sparse and irregular, complicating their attribute compression. While sparse convolution-based methods have made significant success on point cloud attribute compression by leveraging the sparsity of point clouds, they are constrained by a limited receptive field and insufficient adaptability to diverse inputs. To overcome these limitations, this paper proposes a novel point transformer-based architecture to exploit correlations and aggregate features across multiple scales (CSFormer). It retains the advantage of sparse convolution operating on occupied voxels and leverages varied sparsity distributions and the geometry distortions inherent in consecutive scales to construct attention maps, effectively extending the receptive field and adapting to different inputs. We further introduced GCPEM, a Geometry-aware Context Prediction-based Entropy Model that reduces bitrates by jointly utilizing the spatial and channel dependencies. Unlike previous methods that capture only one type of the information, GCPEM organizes latent features into groups interlaced across both space and channel dimensions and employs a context-prediction mechanism guided by known geometry for efficient coding. Experimental results show that the proposed method outperforms the state-of-the-art learning-based method and MPEG standard G-PCC codec over 7% and 28% in BD-BR (Y-PSNR), respectively. It has a time complexity comparable to the state-of-the-art learning-based method and the G-PCC. The source code and trained models will be released at https://github.com/X-H-offical/CST-PCAC-plus.git.

Geometry-based point cloud compression (G-PCC), an international standard designed by MPEG, provides a generic framework for compressing diverse types of point clouds while ensuring interoperability across applications and devices. However, G-PCC underperforms compared to recent deep learning-based PCC methods despite its lower computational power consumption. To enhance the efficiency of G-PCC without sacrificing its interoperability or computational flexibility, we propose the first compression-oriented point cloud voxelization network jointly optimized with a differentiable G-PCC surrogate model. The surrogate model mimics the rate-distortion behavior of the non-differentiable G-PCC codec, enabling end-to-end gradient propagation. The versatile voxelization network adaptively transforms input point clouds using learning-based voxelization and effectively manipulates point clouds via global scaling, fine-grained pruning, and point-level editing for rate-distortion trade-off. During inference, only the lightweight voxelization network is prepended to the G-PCC encoder, requiring no modifications to the decoder, thus introducing no computational overhead for end users. Extensive experiments demonstrate a 38.84% average BD-rate reduction over G-PCC. By bridging classical codecs with deep learning, this work offers a practical pathway to enhance legacy compression standards while preserving their backward compatibility, making it ideal for real-world deployment.

In recent years, LiDAR point clouds have been widely used in many applications. Since the data volume of LiDAR point clouds is very huge, efficient compression is necessary to reduce their storage and transmission costs. However, existing learning-based compression methods do not exploit the inherent angular resolution of LiDAR and ignore the significant differences in the correlation of geometry information at different bitrates. The predictive geometry coding method in the geometry-based point cloud compression (G-PCC) standard uses the inherent angular resolution to predict the azimuth angles. However, it only models a simple linear relationship between the azimuth angles of neighboring points. Moreover, it does not optimize the quantization parameters for residuals on each coordinate axis in the spherical coordinate system. To address these issues, we propose a learning-based predictive coding method (LPCM) with both high-bitrate and low-bitrate coding modes. LPCM converts point clouds into predictive trees using the spherical coordinate system. In high-bitrate coding mode, we use a lightweight Long-Short-Term Memory-based predictive (LSTM-P) module that captures long-term geometry correlations between different coordinates to efficiently predict and compress the elevation angles. In low-bitrate coding mode, where geometry correlation degrades, we introduce a variational radius compression (VRC) module to directly compress the point radii. Then, we analyze why the quantization of spherical coordinates differs from that of Cartesian coordinates and propose a differential evolution (DE)-based quantization parameter selection method, which improves rate-distortion performance without increasing coding time. Experimental results show that LPCM achieved a D1-PSNR BD-rate reduction of 21.2% compared with the G-PCC lossless octree-based coding mode on SemanticKITTI, and 5.6% compared with the PredGeom on Ford, using the latest G-PCC test model TMC13 v31.0.

The rising popularity of cloud gaming highlights the importance of effective video compression for this domain. Despite the potential of neural video codecs to surpass traditional codecs, their application to cloud gaming videos generally achieves limited performance primarily due to two reasons: (1) most advanced neural video codecs are designed for natural videos with only a few optimized for cloud gaming content, and (2) the distinctive characteristics of cloud gaming videos including abrupt and large camera movements, coupled with repetitive textures, make the direct application of existing codecs suboptimal. To this end, this paper introduces an enhanced neural video codec for cloud gaming with camera motion-conditioned motion estimation. By leveraging the unique feature of cloud gaming, we effectively utilize the camera motion information to condition motion estimation with an attention mechanism, overcoming the pronounced challenges of motion estimation and facilitating the accurate learning of optical flow. Furthermore, we optimize the loss function of the motion estimation network by employing a comprehensive loss across multiple feature levels, ensuring the learned optical flow is multi-faceted and well-suited to subsequent motion compensation. Extensive experiments demonstrate the effectiveness of the proposed codec, providing improved motion estimation capabilities and superior rate-distortion performance. Moreover, the robustness of our codec to rapid camera movements is validated, which makes it highly suitable for cloud gaming scenarios.

The metaverse has emerged as a transformative platform that blends physical and digital realities, supporting immersive communication, interaction, and service delivery. At the core of these experiences are avatars, digital embodiments of users, that serve as interactive interfaces across a wide range of metaverse services, including virtual collaboration, education, healthcare, entertainment, e-commerce,etc.However, the fragmented ecosystem of tools, platforms, and proprietary formats has led to significant challenges in avatar interoperability, preventing seamless user identity and experience across applications. In response, numerous international standardization bodies have launched initiatives to define interoperable and modular avatar frameworks. This survey represents the first comprehensive review of these initiatives with a focus on service-oriented requirements, revealing the concept of Avatar-as-a-Service (AaaS). It examines both historical foundations and recent advances, with a particular attention given to the emerging ISO/IEC 23090-39 standard (MPEG-I Part 39 Avatar Representation Format, ARF). By synthesizing technical approaches, standardization efforts, and open challenges, this work presents a roadmap for the evolution of AaaS and highlights emerging opportunities for delivering personalized, high-fidelity, and cross-compatible avatar experiences to support the next generation of metaverse services.

In this paper, we propose a deep hierarchical attention context model for lossless attribute compression of point clouds, leveraging a multi-resolution spatial structure and residual learning. A simple and effective Level of Detail (LoD) structure is introduced to yield a coarse-to-fine representation. To enhance efficiency, points within the same refinement level are encoded in parallel, sharing a common context point group. By hierarchically aggregating information from neighboring points, our attention model learns contextual dependencies across varying scales and densities, enabling comprehensive feature extraction. We also adopt normalization for position coordinates and attributes to achieve scale-invariant compression. Additionally, we segment the point cloud into multiple slices to facilitate parallel processing, further optimizing time complexity. Experimental results demonstrate that the proposed method offers better coding performance than the latest G-PCC for color and reflectance attributes while maintaining more efficient encoding and decoding run-times.

Point clouds serve as a fundamental format for virtual and augmented reality applications. However, their substantial data volumes present considerable challenges regarding data storage and transmission. This paper introduces a novel point transformer-based approach for point cloud attribute compression, termed as CST-PCAC, which utilizes a transformer architecture designed to exploit correlations across multiple scales. Our method is grounded in a variational autoencoder framework, enhanced by stacked sparse convolution (SConv) and the proposed Cross-Scale Point Transformer (CSFormer). CSFormer incorporates two types of attention mechanisms: Within-Scale Attention (WSA) and Cross-Scale Attention (CSA). WSA constructs attention maps within K-nearest neighbours at the current scale, while CSA operates across adjacent upper or lower scales. By leveraging varied sparsity distributions and the geometry distortions inherent in consecutive scales, CSFormer enhances the receptive field and feature representation, thereby effectively extracting correlations to minimize redundancy. Experimental results demonstrate that CST-PCAC achieves average Bjøntegaard Delta bitrate gains exceeding 30%, 10%, and 20% in comparison to Sparse-PCAC, Scalable-PCAC, and G-PCC, respectively. In terms of computational efficiency, CST-PCAC exhibits a time complexity two orders of magnitude lower than that of deep learning-based methods including Sparse-PCAC and Scalable-PCAC, while maintaining a comparable complexity to traditional methods like G-PCC.

Blendshapes have been widely adopted as a key method for generating facial animation on avatars due to their ease of manipulation, flexibility in capturing diverse facial expressions, and compatibility with real-time rendering. However, current frameworks lack efficient methods for compressing blendshape (BS) animation parameters, which are critical for optimizing data transmission. This study introduces a pioneer compression scheme leveraging the amount of BS to be transmitted, their quantization as well as their transmission frequency. Subjective evaluation using the ITU-R BT.500-15 recommendation demonstrates that the proposed method significantly reduces the amount of data to transmit while preserving acceptable visual quality. This approach extends prior findings on reduced BS sets[kang2023effects] and addresses a significant gap in avatar media coding[avril2023morgan]. This work establishes a baseline for efficient facial animation and, serves as a foundation for further exploration on adaptive rate-allocation strategies and advanced compression strategies tailored for diverse avatar animation scenarios.

This paper investigates the impact of standardized Geometry-based Point Cloud Compression (G-PCC) on LiDAR-based object detection tasks. It evaluates how compression distortion types, and distortion levels, together with point cloud density, and object semantics affect various 3D object detection methods, including point-based, voxel-based, and hybrid approaches, using the KITTI object detection dataset.

The rapid advancement of 3D sensing and rendering technologies has expanded the use of point clouds across various fields. To address the challenge of managing large point clouds, Point Cloud Compression (PCC) has gained significant research interest in recent years. The Moving Picture Experts Group (MPEG) has played an important role in standardizing PCC by designing frameworks such as Solid Geometry-based Point Cloud Compression (Solid G-PCC), using octree and Trisoup to compress the geometry of dynamic point clouds. This paper introduces a skip coding mode to improve the inter-frame coding by early terminating octree and Trisoup node generation based on a rate-distortion optimization consideration. Experiments show a 6.3% average improvement in geometry coding efficiency.

Blendshapes (BS) have been widely adopted as a key method for generating facial animations on avatars due to their ease of manipulation, flexibility in capturing diverse expressions, and compatibility with real-time rendering. Despite their importance, current frameworks, including the recent MPEG Avatar Representation Format one, lack efficient methods for blendshape compression. This paper addresses this gap by proposing and analyzing a pioneering compression scheme for blendshape animation parameters. The method explores three key dimensions of compression: the number of transmitted BS, their quantization, as well as their frequency of transmission. The use of a linear interpolation for reduced blendshape transmission rates allow to mitigate flickering introduced by strong quantization, enhancing the viewing experience. Subjective evaluations demonstrate that the proposed approach achieves substantial data transmission savings while maintaining acceptable visual quality. Furthermore, an in-depth comparison of classical objective metrics against Mean Opinion Scores (MOS) reveals their limitations in accurately capturing perceived quality after blendshape compression, with Pearson and Spearman correlation scores reaching at most ~ 0.6. Among the evaluated metrics, Detail Loss Metric (DLM), Video Multimethod Assessment Fusion (VMAF), and Mean Peak Signal-to-Noise Ratio in the BS domain (PSNR-BS) exhibit the highest correlation with MOS. This study provides a comprehensive benchmark for blendshape compression and has driven the creation of an Exploratory Experiment (EE) within the ongoing MPEG avatar-related efforts, highlighting the study’s relevance to standardization activities.

The burgeoning popularity of cloud gaming makes it critical for efficient video compression to relieve the growing bandwidth pressure. While existing neural video coding approaches have demonstrated strong compression potential on natural videos, there is an absence of efficient neural codecs dedicated to gaming videos. To bridge this gap, in this paper, we propose an end-to-end neural video compression method designed specifically for cloud gaming videos. By effectively utilizing the unique camera motion information inherent to cloud gaming, the previous reconstructed frame is maximally aligned to the current frame through a learningbased module with multiple losses, which then replaces the previous reconstructed frame for optical flow estimation. By significantly reducing the displacement between two consecutive frames caused by camera motion, the motion estimation accuracy is enhanced, effectively handling the large and abrupt motion scenarios frequently present in gaming videos. Furthermore, the aligned tensor obtained in the previous step is used to enhance the latent prior of the entropy model, providing a superior temporal prior for coding. Extensive experimental results demonstrate the superior performance of our proposed method compared to one of the previous state-of-the-art approaches, DCVC-HEM, providing significant progress in end-to-end neural compression in cloud gaming videos

Both traditional and learning-based hyperspectral image compression methods suffer from significant quality loss at high compression ratios. To address this, we propose a low-overhead, compression-aware channel filtering method. The encoder derives channel filters via Least Squares Regression between lossy compressed and original images. The bitstream, containing the compressed image and filters, is sent to the decoder, where the filters enhance image quality. This simple, compression-aware approach is compatible with any existing framework, enhancing quality while introducing only a negligible increase in bitstream size and decoding time, thereby achieving low overhead. Experimental results show consistent rate-distortion gains, reducing compression rates by 10.51% to 39.81% on the GF-5 dataset with minimal decoding and storage overhead.

The growing use of hyperspectral images demands efficient compression techniques to handle their extensive spectral data. However, current methods are constrained by their inability to adapt to high bit depth and effectively utilize the spectral characteristics, leading to suboptimal compression ratios. This paper presents a novel hyperspectral compression framework that employs a lightweight spectral super-resolution network to address these limitations. The proposed approach divides the hyperspectral image into two sub-images, comprising two distinct groups of bands: a base image consisting of anchor bands and a supplementary image comprising non-anchor bands. The base image is compressed losslessly using a conventional codec, thereby ensuring the preservation of essential information. In contrast, the supplementary image is compressed efficiently by overfitting a lightweight super-resolution network to predict the non-anchor bands during encoding. The optimized network parameters are encoded as side information to ensure high-quality spectral super-resolution during decoding. Experimental results on the ARAD hyperspectral image dataset demonstrate that our approach significantly outperforms state-of-the-art methods, effectively meeting the demand for efficient hyperspectral image compression while maintaining acceptable processing speeds.

Dynamic point cloud compression is essential for efficient 3D visual data transmission and storage. To achieve satisfactory coding efficiency, existing learning-based frameworks typically design separate networks for intra-frame and inter frame geometry compression, resulting in frequent model switching, buffering, and loading, which increases bus bandwidth consumption and latency within hardware systems. Our experiments show that model switching can account for 30.4% of per-frame coding time, especially in longer sequences with fewer frames per GOF (Group of Frames). To overcome this, this paper proposes a unified geometry compression framework that handles both intra-frame and inter-frame coding within a single network. Central to our approach is a 4D sparse tensor representation that enables the joint extraction of spatial and temporal features. We further enhance compression with two modules: cross-scale feature aggregation (CFA), which improves spatial representation by fusing coarse and fine features; and bidirectional feature propagation (BiFP), which boosts temporal modelling by enabling mutual feature propagation across frames. Together, these modules exploit spatiotemporal correlations for improved efficiency. Extensive experiments demonstrate that our method consistently outperforms state-of-the-art baselines across various datasets.

Three-dimensional (3D) point clouds are becoming more and more popular for representing 3D objects and scenes. Due to limited network bandwidth, efficient compression of 3D point clouds is crucial. To tackle this challenge, the Moving Picture Experts Group (MPEG) is actively developing the Geometry-based Point Cloud Compression (G-PCC) standard, incorporating innovative methods to optimize compression, such as the Region-Adaptive Hierarchical Transform (RAHT) nestled within a layer-by-layer octree-tree structure. Nevertheless, a notable problem still exists in RAHT, i.e., the proportion of zero residuals in the last few RAHT layers leads to unnecessary bitrate consumption. To address this problem, we propose an adaptive skip coding method for RAHT, which adaptively determines whether to encode the residuals of the last several layers or not, thereby improving the coding efficiency. In addition, we propose a rate-distortion cost calculation method associated with an adaptive Lagrange multiplier. Experimental results demonstrate that the proposed method achieves average Bjøntegaard rate improvements of -3.50%, -5.56%, and -4.18% for the Luma, Cb, and Cr components, respectively, on dynamic point clouds, when compared with the state-of-the-art G-PCC reference software under the common test conditions recommended by MPEG.

The continuous evolution of cloud gaming makes it crucial for efficient video compression to relieve the growing bandwidth pressure. This paper presents a coding optimization algorithm for cloud gaming videos based on three-dimensional image warping, considering the significance of reference frames in inter-frame prediction. By leveraging camera parameters and motion information from the game engine, we generate a virtual reference frame through three-dimensional image warping on the selected key frame and the coloring operation. The resulting virtual reference frame exhibits minimal spatial differences from the frame to be encoded and is utilized to update the original reference frame list in two different methods, achieving accurate inter-frame prediction and significantly reducing coding residuals. Experimental results demonstrate the superiority of our algorithm compared to the mainstream cloud gaming encoder in various camera movements, achieving an average BD-rate reduction of 5.60% and BD-PSNR gain of 0.55dB.

We present an approach for compressing point cloud geometry by leveraging a lightweight super-resolution network. It involves decomposing a point cloud into a base point cloud and the interpolation patterns for reconstructing the original point cloud. While the base point cloud can be efficiently compressed using any lossless codec, such as Geometry-based Point Cloud Compression, a distinct strategy is employed for handling the interpolation patterns. Rather than directly compressing the interpolation patterns, a lightweight super-resolution network is utilized to learn this information through overfitting. Subsequently, the network parameter is transmitted to assist in point cloud reconstruction at the decoder side. Our approach differentiates itself from lookup table-based methods, allowing us to obtain more accurate interpolation patterns by accessing a broader range of neighboring voxels at an acceptable computational cost. Experiments on MPEG Cat1 (Solid) and Cat2 datasets demonstrate the remarkable compression performance achieved by our method.

Geometry-based point cloud compression (G-PCC) uses Context-based Adaptive Binary Arithmetic Coding to encode the geometry and attribute information. The context information is built in context models for entropy coding. G-PCC adopts the Optimal Binarization with Update On-the-fly (OBUF) to reduce the number of context models. In the current design, however, the probability initialization for both fine-and coarse-grained contexts does not follow the principle of entropy continuation. Moreover, the mapping process to produce coarse-grained contexts is a combination of several fine-grained contexts, leading to an unstable update of probability for coarse-grained contexts, which affects the accuracy of the fine-grained context model in probability estimation.To address the underlying problems, we propose two approaches to improve OBUF: initializing the probabilities for fine-grained and coarse-grained contexts according to entropy continuation and setting the probability update upper and lower bounds for coarse-grained contexts adaptively. The experimental results demonstrate that the proposed technique is more consistent with the underlying principles of OBUF and significantly improves the performance of both octree-based and Trisoup-based geometry coding. Due to the theoretical consistency and outstanding performance, the proposed methods have been adopted into the state-of-the-art G-PCC.

Point clouds, crucial for representing 3D objects and scenes, offer immersive and precise depictions of the real world. Despite their superiority, the substantial data volume challenges current multimedia ecosystems. To address this, the Moving Picture Expert Group (MPEG) initiated the point cloud compression project in 2017, leading to two branches: Video-based Point Cloud Compression (V-PCC) and Geometry-based Point Cloud Compression (G-PCC). The first edition of G-PCC was published in March 2023, and ongoing efforts over the past three years have advanced towards G-PCC Edition 2. This paper aims to present recent technical achievements and the current status of G-PCC standardization activities.

Versatile Video Coding (H.266/VVC) significantly enhances compression performance but also increases encoding complexity. Numerous fast intra coding algorithms have been proposed to balance the performance gains with the increased complexity introduced by various advanced tools. However, few fast algorithms are specific to Joint Coding of Chroma Residuals (JCCR), a newly introduced technique that requires additional rate-distortion optimization (RDO) processes. To address this gap, this letter introduces a fast intra coding algorithm that reduces complexity by early skipping unnecessary JCCR. Specifically, we propose skipping JCCR when the correlation between chroma components is low, and we construct a JCCR normalized reconstruction distortion to measure this correlation. The skip conditions are determined by statistical analysis, intelligently reducing the number of RDO processes for chroma components. Experimental results show that, compared to the Fraunhofer Versatile Video Encoder (VVenC), our algorithm achieves encoding time savings of 1.68% and 2.82% under Random Access (RA) and All Intra (AI) settings, respectively, with a performance loss of 0.03% and 0.09% only, demonstrating the effectiveness of our fast algorithm.

The new generation video coding standard, H.266/-VVC, brings performance gains while significantly increasing coding complexity. Most of the complexity optimization methods for intra coding focus on reducing the number of prediction modes entering the rate-distortion optimization, but there are few concerning the newly introduced secondary transform. This letter extends the scope of existing studies and provides valuable insights for intra coding complexity optimization based on early skipping of the secondary transform. Specifically, considering the distribution of residuals assessed by coefficients after the primary transform, we propose to skip the secondary transform when the residual variation is relatively flat or the distortion caused by zeroing high-frequency coefficients is significant. Based on this analysis, we establish two skip conditions through statistical analysis and enable efficient decision-making by skipping the unnecessary secondary transform. Experimental results demonstrate the effectiveness of our algorithm, achieving 3.84% and 10.29% reductions in encoding time with only 0.39% and 1.03% increases in Bjontegaard delta bit-rate compared to Fraunhofer Versatile Video Encoder (VVenC)’s medium preset, under random-access (RA) and all-intra (AI) configurations respectively.

Due to the increasing demand for game-streaming services, efficient compression of computer-generated video is more critical than ever, especially when the available bandwidth is low. This paper proposes a super-resolution framework that improves the coding efficiency of computer-generated gaming videos at low bitrates. Most state-of-the-art super-resolution networks generalize over a variety of RGB inputs and use a unified network architecture for frames of different levels of degradation, leading to high complexity and redundancy. Since games usually consist of a limited number of fixed scenarios, we specialize one model for each scenario and assign appropriate network capacities for different QPs to perform super-resolution under the guidance of reconstructed high-quality luma components. Experimental results show that our framework achieves a superior quality-complexity trade-off compared to the ESRnet baseline, saving at most 93.59% parameters while maintaining comparable performance. The compression efficiency compared to HEVC is also improved by more than 17% BD-rate gain.

Due to the ability to depict large-scale 3D scenes, point clouds acquired by the Light Detection And Ranging (LiDAR) devices have played an indispensable role in various fields. The growing data amount of point cloud, however, brings huge challenges to existing point cloud processing networks. Developing point cloud compression algorithms has become an active research area in recent years. Representative compression frameworks include the MPEG Geometry-based Point Cloud Compression (G-PCC) standard in which a dedicated profile is designed for spinning LiDAR point clouds. In that design, prior knowledge of the LiDAR device is used to project points to nodes in a predictive structure which better reflects the spatial correlation of LiDAR point clouds. In this paper, an analysis has been conducted to explain the observed irregular point distribution in the predictive structure. A regularized projection algorithm is then proposed to construct a reliable prediction relationship in the predictive structure. Simplified geometry prediction techniques are further proposed based on the regularized projection pattern. Experimental results show that an average BD-rate gain of 18% can be achieved with lower encoding runtime if compared with MPEG G-PCC.

Point cloud is a major representation format of 3D objects and scenes. It has been increasingly applied in various applications due to the rapid advances in 3D sensing and rendering technologies. In the field of autonomous driving, point clouds captured by spinning Light Detection And Ranging (LiDAR) devices have become an informative data source for road environment perception and intelligent vehicle control. On the other hand, the massive data volume of point clouds also brings huge challenges to point cloud transmission and storage. Therefore, establishing compression frameworks and algorithms that conform to the characteristics of point cloud data has become an important research topic for both academia and industry. In this paper, a geometry compression method dedicated to spinning LiDAR point cloud was proposed taking advantage of the prior information of the LiDAR acquisition procedure. Rate-distortion optimizations were further integrated into the coding pipeline according to the characteristics of the prediction residuals. Experimental results obtained on different datasets show that the proposed method consistently outperforms the state-of-the-art G-PCC predictive geometry coding method with reduced runtime at both the encoder and decoder sides.

Point cloud is an informative type of media to represent objects and scenes. Along with the geometry information that records the 3D shapes, multiple types of attribute information can be included to characterize the visual appearance of objects and scenes. To deal with the massive data volume of point clouds, developing compression algorithms is necessary. While various prediction techniques have been proposed to reduce the information redundancy for individual point cloud attribute, the correlation between different types of attribute is, however, not fully investigated. This paper first studies the correlation between different types of attribute. A cross-type attribute prediction method is then proposed. The proposed method is evaluated on top of different point cloud compression frameworks with various point cloud sequences. Experimental results show that the proposed cross-type attribute prediction method is able to significantly improve the coding efficiency with marginal coding complexity.

With the rapid development of 3D sensing and reconstruction technologies, point cloud compression has become an active research area. When designing compression methods for point cloud attributes, the reconstructed geometry information is commonly used to guide the attribute compression. Specifically, the geometric distance between points is widely applied in different forms to assist attribute prediction. However, due to the complicated spatial distributions of 3D point clouds, using distance as the sole feature may not be optimal. In this paper, the spatial distribution of point clouds is further considered to derive a distribution-driven attribute pre-diction. Experimental results show that the proposed method can significantly improve the coding efficiency of G-PCC. The proposed method has been adopted in the technologies under consideration (TuC) for MPEG G-PCC and will be included in the second version of the G-PCC specification.

Point clouds captured by Light Detection And Ranging (Li-DAR) devices have played a significant role in autonomous driving and high-precision mapping. The massive amount of point cloud data, however, challenges the capacity of current data storage and transmission networks, which confines the development of LiDAR point cloud applications. To alleviate this situation, a novel grid-based geometry compression framework dedicated to spinning LiDAR point cloud is proposed in this paper. Firstly, a 2D grid-based point cloud representation is built taking advantage of the LiDAR acquisition pattern. Then, a projection is performed to effectively represent the 3D geometry as multiple 2D geometry components. Finally, dedicated prediction and entropy coding methods are designed for each 2D geometry component according to its characteristics. Experimental results show that the proposed method outperforms MPEG G-PCC with an average gain of 18.81% and 8.03% for lossy and lossless coding respectively.

Point clouds captured by spinning Light Detection And Ranging (LiDAR) devices have played a significant role in many applications. To efficiently store and transmit such a huge amount of data, MPEG designed the Geometry-based Point Cloud Compression (G-PCC) standard, which includes a dedicated Pre-dictive Profile for LiDAR point clouds. In this scheme, Cartesian coordinates are mapped to the spherical representation using the elevation-related LiDAR calibration parameters to better characterize the spherical acquisition pattern of the spinning LiDAR device. As such, stronger spatial correlations exist among neighbouring nodes in the predictive structure, resulting in higher compression efficiency. However, it should be mentioned that the azimuth-related calibration parameters, which are unused, also impact the accuracy and correctness of the mapped spherical coordinates. In this paper, an azimuth adjustment method is proposed taking into account this impact. Experimental results show that the proposed azimuth adjustment can consistently improve the coding efficiency of G-PCC. Furthermore, a LiDAR calibration parameter estimation method is proposed in case the azimuth-related parameters are absent. Results show that the proposed calibration parameter estimation can precisely approximate the ground truth.

Parametric models of video quality are designed for service and network planning as well as video quality monitoring. They are widely applied in a broad range of applications especially when video streams are encrypted or even unavailable at all. Designing metrics in these models remains challenge due to limited available information. In this paper, a spatio-temporal resolution-adaptive parametric (STRAP) model is proposed to evaluate the quality of video streaming services considering the spatial and temporal resolutions. This work serves as a follow-up study for ITU Rec. P.1203.1 that we were previously involved. The relationship between the content complexity and the spatial and temporal resolutions are analyzed and incorporated into the proposed model. Moreover, the effect of video up/down-scaling in display devices on the perceived video quality is further taken into consideration. Experimental results showed that the proposed model can be used as a reliable indicator for video streaming providers to improve their services performance.

Instead of streaming the entire omnidirectional video that induces a waste of bandwidth, the tile-based viewport adaptive streaming is preferred in practice. Tiles within users’ viewport are transmitted in high quality, while tiles elsewhere are delivered in lower quality to reduce the network bandwidth consumption. In this setup, when users change their viewport, they may potentially be disturbed by the exposure of low-quality tiles and the subsequent quality refinement in a period of delay. We defined this phenomenon in the viewport as a Dynamic Quality Boundary (DQB) artifact, as there exists a clear boundary between low-quality tiles and high-quality tiles which appears, moves, and disappears along with the change of users’ viewport. The DQB artifact is a specific quality degradation in the viewport-adaptive omnidirectional video streaming whose impact on users’ perceptual quality has not been adequately studied. In this paper, we focused on investigating and modeling its impact on the quality perception of users when streaming omnidirectional videos. More specifically, systematic subjective quality evaluation experiments towards DQB artifact were designed and conducted. Accordingly, a perceptual quality metric was proposed considering the quality of tiles, the proportion of low-quality tiles, and the refinement duration. Experimental results, measured in various aspects, showed that the proposed metric can accurately predict users’ perceptual quality of the tile-based viewport-adaptive omnidirectional video streaming service.

With the development of multimedia technologies and virtual reality display devices, omnidirectional videos have gained popularity nowadays. To reduce the bandwidth requirement for omnidirectional video transmission, tile-based viewport adaptive streaming methods have been proposed in the literatures. Challenges related to decoding the tiles simultaneously with limited number of decoders, and ensuring user's viewing experience during the viewport switch are still to be solved. In this paper, a two-layer fast viewport switching dynamic adaptive streaming over HTTP (DASH) system based on tiling mechanism is proposed, which incorporates the viewing trajectory of end users. To deal with the simultaneously decoding problem, an open group of picture (GOP) technique is proposed to enable merging different types of tiles into a composite stream at the client side. To reduce the quality recovery duration after the viewport change, a fast-switching strategy is also proposed. Moreover, considering the priorities of different types of chunks, a download strategy is further proposed to adapt the bandwidth fluctuations and viewport changes. Experimental results showed that the proposed system can significantly reduce the recovery duration of high quality video by approximately 90%, which can provide a better viewing experience to end users.

Recent years have witnessed increasing advances in immersive media technologies. Omnidirectional videos, which record a 360-degree spherical visual scene, have gained popularity at an accelerated speed. However, streaming this visual media is extremely bandwidth demanding when compared to traditional 2D videos, thus challenging current video streaming strategies and network conditions. Understanding how human visual system perceives the omnidirectional video becomes an urgent research topic. In this work, we focus on investigating the impact of stalling on the perceptual quality of omnidirectional video streaming. A systematic experiment with various phases was conducted to obtain the quality perception results towards omnidirectional video stalling. Experimental results showed that human visual system views the omnidirectional video in a different manner if compared with traditional 2D videos. This difference further triggers an alleviation in the annoyance level of stalling. A questionnaire focused on revealing the underlying reason regarding this difference was further designed. Findings of this work provide insights into the visual quality perception of stalling events during omnidirectional video playback. Conclusions of this paper can be used as guidelines for the design of video streaming protocols as well as network dimensioning.

With the development of immersive media technologies, omnidirectional video services have been launched in many fields. Conducting subjective quality evaluation research becomes a crucial step to benchmark and ensure the quality of omnidirectional video services. As omnidirectional videos record spherical visual scenes that are broader than the visual field of human eyes, the quality scores rated by different observers are based on individual spatio-temporal viewing experience. The potential spatio-temporal inconsistency between observers may impact the reliability of subjective quality evaluation and thus challenge existing experimental methodologies. In this paper, we focus on investigating the effect of spatial-temporal inconsistency on the subjective quality evaluation of omnidirectional videos. A systematic quality evaluation experiment was designed with various viewing methods involved. Experimental results showed that the spatio-temporal inconsistency has a significant impact on the reliability of subjective quality results and the impact is strongly determined by the viewing method. We intend to provide recommendations with respect to the subjective quality evaluation of omnidirectional videos.

Our previous study has shown that image distortions cause saliency distraction, and that visual saliency of a distorted image differs from that of its distortion-free reference. Being able to measure such distortion-induced saliency variation (DSV) significantly benefits algorithms for automated image quality assessment. Methods of quantifying DSV, however, remain unexplored due to the lack of a benchmark. In this paper, we build a benchmark for the measurement of DSV through a subjective study. Sixteen experts in computer vision were asked to compare saliency maps of distorted images to the corresponding saliency maps of the original images. All saliency maps were rendered from ground truth human fixations. A statistical analysis is performed to reveal the behaviours and properties of human assessment of the saliency variation. The benchmark is made publicly available to the research community.

Point cloud is a main representation of 3D scenes. It is widely applied in many fields including autonomous driving, heritage reconstruction, virtual reality and augmented reality. The data size of this type of media is massive since it contains numerous points with each associated with a large amount of information including geometric coordinate, color, reflectance, and normal. It is thus of great significance to investigate the compression of point cloud data to boost its application. However, developing efficient point cloud compression method is challenging mainly due to the unstructured nature and nonuniform distribution of the data. In this paper, we propose a novel point cloud attribute compression algorithm based on Haar Wavelet Transform (HWT). More specifically, the transform is performed taking into account the surface orientation of point cloud. Experimental results demonstrate that the proposed method outperforms other state-of-the-art transforms.

Telemedicine can provide timely and high-quality clinical health care from a distance, improving access to and delivery of medical services in resource-poor settings. It can also save lives in situations of emergency. In many circumstances, the success of telemedicine practice heavily relies on the transmission of medical videos over large distances. However, video communication systems are prone to distortion in visual signals, affecting the task performance and thus putting patients at risk. It is critical to understand how practitioners perceive the quality of visual media and use such knowledge to improve clinical practice in telemedicine. In this paper, we investigate the hypothesis that visual quality perception varies between clinicians who work in different practice settings. To evaluate this hypothesis, we performed a subjective experiment where French and Chinese radiologists were asked to rate the quality of ultrasound videos compressed using different compression configurations. The results show that the way the perceived quality changes with the compression configuration is consistent among the both settings studied, however, French radiologists were more bothered by the compression artifacts. The findings can help inform future studies to develop tailored telemedicine systems for specific settings or individuals.

Objective image quality metrics (IQMs) potentially benefit from the addition of visual saliency. However, challenges to optimizing the performance of saliency-based IQMs remain. A previous eye-tracking study has shown that gaze is concentrated in fewer places in images with highly salient features than in images lacking salient features. From this, it can be inferred that the former are more likely to benefit from adding a saliency term to an IQM. To understand whether these ideas still hold when using computational saliency instead of eye-tracking data, we first conducted a statistical evaluation using 15 state-of-the-art saliency models and 10 well-known IQMs. We then used the results to devise an algorithm, which adaptively incorporates saliency in IQMs for natural scenes, based on saliency dispersion. Experimental results demonstrate that this can give significant improvements.

The popularity of HTTP-based video streaming services has been increasing in recent years. The quality of HTTP-based video streaming services is measured by the user's quality of experience (QoE). Perceptual quality (PQ), a strong indicator of the QoE, has been widely studied in the literature. Specifically, previous studies primarily focused on modeling the overall perceptual quality at the end of the viewing process. The specific influence of each event during the viewing process was neglected. Since the PQ is a function of time, which models the human perception of audiovisual services, this contribution focuses on investigating the time-varying feature of human perception. In this paper, the viewing timeline of subjects is divided into rebuffering and playback. An event-based perceptual quality assessment (EPQ) framework is introduced, which can assess the PQ of interrupted HTTP-based video streaming at any point in time. To understand the human perception of video streams with playback interruptions, a subjective quality assessment experiment was designed to obtain the PQ at a series of points in time. A between-subjects design was adopted in which each subject can view video content without repetition. Based on the experimental results, a PQ assessment model was proposed to predict the change in PQ (i.e., ΔPQ) after each rebuffering and playback. The overall PQ at any point in time is calculated as the summation of the ΔPQs of all the previous events. The experimental results demonstrated that the EPQ-based model outperforms the rebuffering components contained in the ITU-T Rec. P.1201.1, P.1201 Amd. 2 and P.1203.3 in our database.

Reliably predicting video quality as perceived by humans remains challenging and is of high practical relevance. A significant research trend is to investigate visual saliency and its implications for video quality assessment. Fundamental problems regarding how to acquire reliable eye-tracking data for the purpose of video quality research and how saliency should be incorporated in objective video quality metrics (VQMs) are largely unsolved. In this paper, we propose a refined methodology for reliably collecting eye-tracking data, which essentially eliminates bias induced by each subject having to view multiple variations of the same scene in a conventional experiment. We performed a large-scale eye-tracking experiment that involved 160 human observers and 160 video stimuli distorted with different distortion types at various degradation levels. The measured saliency was integrated into several best known VQMs in the literature. With the assurance of the reliability of the saliency data, we thoroughly assessed the capabilities of saliency in improving the performance of VQMs, and devised a novel approach for optimal use of saliency in VQMs. We also evaluated to what extent the state-of-the-art computational saliency models can improve VQMs in comparison to the improvement achieved by using "ground truth" eye-tracking data. The eye-tracking database is made publicly available to the research community.

Image quality assessment potentially benefits from the addition of visual attention. However, incorporating aspects of visual attention in image quality models by means of a perceptually optimized strategy is largely unexplored. Fundamental challenges, such as how visual attention is affected by the concurrence of visual signals and their distortions; whether visual attention affected by distortion or that driven by the original scene only should be included in an image quality model; and how to select visual attention models for the image quality application context, remain. To shed light on the above unsolved issues, designing and performing eye-tracking experiments are essential. Collecting eye-tracking data for the purpose of image quality study is so far confronted with a bias due to the involvement of stimulus repetition. In this paper, we propose a new experimental methodology to eliminate such inherent bias. This allows obtaining reliable eye-tracking data with a large degree of stimulus variability. In fact, we first conducted 5760 eye movement trials that included 160 human observers freely viewing 288 images of varying quality. We then made use of the resulting eye-tracking data to provide insights into the optimal use of visual attention in image quality research. The new eye-tracking data are made publicly available to the research community.

A significant current research trend in image quality assessment is to investigate the added value of visual attention aspects. Previous approaches mainly focused on adopting a specific saliency model to improve a specific image quality metric (IQM). It is still not known yet which of the existing saliency models is generally applicable in IQMs; which of the IQMs can profit most/least from the addition of saliency; and how this improvement depends on the saliency model used and the IQM targeted. In this paper, a large-scale benchmark study is conducted to assess the capabilities and limitations of the state-of-the-art saliency models in the context of IQMs. The study provides guidance for the application of saliency models in IQMs, in terms of the effect of saliency model dependency, IQM dependency, and image distortion dependency.

Finding ways to be able to objectively and reliably assess video quality as would be perceived by humans has become a pressing concern in the multimedia community. To enhance the performance of video quality metrics (VQMs), a research trend is to incorporate visual saliency aspects. Existing approaches have focused on utilizing a computational saliency model to improve a VQM. Since saliency models still remain limited in predicting where people look in videos, the benefits of inclusion of saliency in VQMs may heavily depend on the accuracy of the saliency model used. To gain an insight into the actual added value of saliency in VQMs, ground truth saliency obtained from eye-tracking instead of computational saliency is an essential prerequisite. However, collecting eye-tracking data within the context of video quality is confronted with a bias due to the involvement of massive stimulus repetition. In this paper, we introduce a new experimental methodology to alleviate such potential bias and consequently, to be able to deliver reliable intended data. We recorded eye movements from 160 human observers while they freely viewed 160 video stimuli distorted with different distortion types at various degradation levels. We analyse the extent to which ground truth saliency as well as computational saliency actually benefit existing state of the art VQMs. Our dataset opens new challenges for saliency modelling in video quality research and helps better gauge progress in developing saliency-based VQMs.

Saliency modelling for image quality research has been an active topic in multimedia over the last five years. Saliency aspects have been added to many image quality metrics (IQMs) to improve their performance in predicting perceived quality. However, challenges to optimising the performance of saliency-based IQMs remain. To make further progress, a better understanding of human attention deployment in relation to image quality through eye-tracking experimentation is indispensable. Collecting substantial eye-tracking data is often confronted with a bias due to the involvement of massive stimulus repetition that typically occurs in an image quality study. To mitigate this problem, we proposed a new experimental methodology with dedicated control mechanisms, which allows collecting more reliable eye-tracking data. We recorded 5760 trials of eye movements from 160 human observers. Our dataset consists of 288 images representing a large degree of variability in terms of scene content, distortion type as well as degradation level. We illustrate how saliency is affected by the variations of image quality. We also compare state of the art saliency models in terms of predicting where people look in both original and distorted scenes. Our dataset helps investigate the actual role saliency plays in judging image quality, and provides a benchmark for gauging saliency models in the context of image quality.

Advances in image quality assessment have shown the potential added value of including visual attention aspects in its objective assessment. Numerous models of visual saliency are implemented and integrated in different image quality metrics (IQMs), but the gain in reliability of the resulting IQMs varies to a large extent. The causes and the trends of this variation would be highly beneficial for further improvement of IQMs, but are not fully understood. In this paper, an exhaustive statistical evaluation is conducted to justify the added value of computational saliency in objective image quality assessment, using 20 state-of-the-art saliency models and 12 best-known IQMs. Quantitative results show that the difference in predicting human fixations between saliency models is sufficient to yield a significant difference in performance gain when adding these saliency models to IQMs. However, surprisingly, the extent to which an IQM can profit from adding a saliency model does not appear to have direct relevance to how well this saliency model can predict human fixations. Our statistical analysis provides useful guidance for applying saliency models in IQMs, in terms of the effect of saliency model dependence, IQM dependence, and image distortion dependence. The testbed and software are made publicly available to the research community.

Advances in video quality assessment have shown the added value of including temporal aspects of artifact perception in its objective metrics. The impact of quality variations over time and the associated implications for the assessment of overall video quality are so far not fully understood yet. To investigate the human behavioural responses to time-varying distortions and the relevance of such perception to overall quality judgements, a series of subjective experiments were conducted. In the major experiment, 30 human subjects scored the overall quality of 120 stimuli, which consist of 84 videos of various time-varying quality profiles and of their corresponding 36 constituent segments of spatio-temporally constant quality. Results show that the pattern of quality variations over time tends to affect the temporal summation strategy that delivers an overall quality.

Novel research on image quality metrics (IQMs) attempts to further improve their reliability by including visual attention aspects of the human visual system. Literature so far mainly focuses on the extension of a specific IQM with a specific visual saliency model. In this paper, we quest the integration of visual saliency models in IQMs, in terms of its statistical meaningfulness and combination strategy. In the first step an exhaustive evaluation is conducted by integrating twenty state-of-the-art saliency models into eight best-known IQMs for image quality assessment. It demonstrates linearly combining saliency and IQMs yields a statistically significant gain in performance. Based on the statistics, we revisit the combination strategy of saliency and IQMs and propose a new strategy taking into account the distraction power of local distortions. Results show that the proposed combination strategy consistently outperforms the conventionally used linear combination strategy.

Advances in image quality assessment have shown the potential added value of including visual attention aspects in objective quality metrics. Numerous models of visual saliency are implemented and integrated in different quality metrics; however, their ability of improving a metric's performance in predicting perceived image quality is not fully investigated. In this paper, we conduct an exhaustive comparison of 20 state-of-the-art saliency models in the context of image quality assessment. Experimental results show that adding computational saliency is beneficial to quality prediction in general terms. However, the amount of performance gain that can be obtained by adding saliency in quality metrics highly depends on the saliency model and on the metric.