Attention models, particularly Transformers, have significantly advanced deep learning in fields like natural language processing and computer vision by capturing contextual relationships in both sequential and spatial data. This ability is valuable for Point Clouds (PC), which are unstructured sets of points in 3D space. Transformers can effectively identify correlations between distant points, allowing them to focus on the most critical regions of the data. To demonstrate this capability, this paper proposes a novel, scalable Graph-Guided Transformer model, labeled 2GFormer, for static PC geometry. This model is built using a scalable architecture that leverages Graph Convolutions to enhance a Relational Neighborhood SelfAttention (RNSA) base layer model. Both models are integrated into the JPEG Pleno Learning-based Point Cloud Coding (JPEG PCC) standard, resulting in the creation of two attention-enabled codecs for static PC coding: JPEG RNSA and JPEG 2GFormer. While JPEG RNSA codec delivers significant compression improvements for solid and dense PCs compared to the baseline JPEG PCC standard, JPEG 2GFormer extends these gains to solid, dense, and sparse PCs with only a marginal increase in model parameters. Additionally, JPEG 2GFormer outperforms both conventional and learning-based state-of-the-art PC codecs. These results position JPEG 2GFormer as a highly efficient solution for versatile PC coding.

In this golden age of multimedia, realistic content is in high demand with users seeking more immersive and interactive experiences. As a result, new image modalities for 3D representations have emerged in recent years, among which point clouds have deserved especial attention. Naturally, with this increase in demand, efficient storage and transmission became a must, with standardization groups such as MPEG and JPEG entering the scene, as it happened before with other types of visual media. In a surprising development, JPEG issued a Call for Proposals on point cloud coding targeting exclusively learning-based solutions, in parallel to a similar call for image coding. This is a natural consequence of the growing popularity of deep learning, which due to its excellent performances is currently dominant in the multimedia processing field, including coding. This paper presents the coding solution selected by JPEG as the best-performing response to the Call for Proposals and adopted as the first version of the JPEG Pleno Point Cloud Coding Verification Model, in practice the first step for developing a standard. The proposed solution offers a novel joint geometry and color approach for point cloud coding, in which a single deep learning model processes both geometry and color simultaneously. To maximize the RD performance for a large range of point clouds, the proposed solution uses down-sampling and learning-based super-resolution as pre- and post-processing steps. Compared to the MPEG point cloud coding standards, the proposed coding solution comfortably outperforms G-PCC, for both geometry, color, and joint quality metrics.

Point Clouds (PCs) have gained significant attention due to their usage in diverse application domains, notably virtual and augmented reality. While PCs excel in providing detailed 3D visualization, this typically requires millions of points which must be efficiently coded for real-world deployment, notably storage and streaming. Recently, learning-based coding solutions have been adopted, notably in the JPEG Pleno Point Coding (PCC) standard, which uses a coding model with millions of model parameters. This requires the use of high-performance computing devices, which may not be available, notably at the decoder side. In this context, this paper proposes two reduced complexity decoding solutions, based on the adoption of scalability principles, to decode the same JPEG PCC compliant bitstreams. The design of these solutions is based on two innovative model pruning strategies which reduce the decoding complexity. The experimental results demonstrate the effective capability to significantly reduce the number of decoding model parameters with an acceptable penalty on Rate-Distortion (RD) performance compared to the full complexity model.

The widespread usage of point clouds (PC) for immersive visual applications has resulted in the use of very heterogeneous receiving conditions and devices, notably in terms of network, hardware, and display capabilities. In this scenario, quality scalability, i.e., the ability to reconstruct a signal at different qualities by progressively decoding a single bitstream, is a major requirement that has yet to be conveniently addressed, notably in most learning-based PC coding solutions. This paper proposes a quality scalability scheme, named Scalable Quality Hyperprior (SQH), adaptable to learning-based static point cloud geometry codecs, which uses a Quality-conditioned Latents Probability Estimator (QuLPE) to decode a high-quality version of a PC learning-based representation, based on an available lower quality base layer. SQH is integrated in the future JPEG PC coding standard, allowing to create a layered bitstream that can be used to progressively decode the PC geometry with increasing quality and fidelity. Experimental results show that SQH offers the quality scalability feature with very limited or no compression performance penalty at all when compared with the corresponding non-scalable solution, thus preserving the significant compression gains over other state-of-the-art PC codecs.

In the ever-evolving landscape of deep learning, attention models have contributed to boost the performance in diverse fields such as computer vision and natural language processing. Following this trend, this paper proposes a novel Relational Neighborhood Self-Attention (RNSA) model, specifically designed for Point Cloud (PC) geometry coding to be integrated in the emerging learning-based JPEG PCC standard. The RNSA model proposes three new methods: first, to effectively learn correlations between the points by capturing the relational features and positions of neighboring points; second, to address the inefficiencies of conventional dot product attention, a novel Relational Scoring method to generate an attention map able to capture both linear and non-linear relationships between points and their neighbors is adopted; third, the created attention maps are normalized by Sparsemax instead of Softmax to generate sparse probabilities and assigns higher scores to the most important neighbors while marginalizing the less significant ones. Experimental results show that the proposed attention model achieves around 8% gains in both BD-Rate PSNR Dl and PSNR D2 compared to the baseline codec, i.e., JPEG PCC, while adding a small number of model parameters to JPEG PCC.

Multimedia applications have been evolving towards providing users with more immersive and realistic experiences. A common way to model the light available for the users’ eyes is the so-called plenoptic function – a powerful 7D representation of light. There are three main types of 3D representation models for the plenoptic function, capable of expressing the light information needed to offer 6-Degrees of Freedom (DoF) experiences, namely light fields, meshes, and Point Clouds (PCs). This paper focuses on PCs since they allow representing and processing objects directly in the 3D space, facilitating user interaction and navigation in a multitude of application domains. Since the illusion of real surfaces is provided by high-density point sets, a good quality of experience requires a rather large set of points to represent a single PC, thus originating huge amounts of data to be stored and/or transmitted. Consequently, PC Coding (PCC) with significant compression levels is a must to reduce the PC data to more manageable sizes and bring PC-based applications to practical deployment. The promising results for image coding led the Joint Photographic Experts Group (JPEG) to launch a standardization project especially targeting Deep Learning (DL)-based PCC, with a final Call for Proposals in January 2022. The best performing response to this call [1] became the JPEG Pleno Learning-based PCC Verification Model (VM), which is the seed codec for the final standard. In this codec, the rate may be controlled through a set of coding parameters, largely depending on the specific PC to code, notably its sparsity and homogeneity.

Despite its novelty, learning-based coding for images and point clouds is already outperforming some of the best long-standing conventional codecs. In addition to its rising compression performance, learning-based coding has opened new opportunities, notably the use of a single compressed domain representation to provide both high fidelity reconstructions for human visualization as well as effective performance for computer vision tasks, effectively unifying the visual language for man and machine. This paper proposes a new double learning-based static point cloud geometry and color coding solution, which targets point cloud component scalability, rate control flexibility at coding time, and a unified compressed domain representation. The proposed solution exploits the synergies between learning-based coding for images and point clouds, through the current JPEG PCC standard for geometry coding and the JPEG AI standard for image/color coding, establishing a learning-based ecosystem for JPEG coding standards. The proposed solution is able to overcome some design limitations of the current JPEG PCC Verification Model, and significantly improve its RD performance, becoming competitive with MPEG PCC standards.

Point Clouds represent one of the most versatile 3D visual representation models as they can provide the user the six degrees of freedom required for a truly immersive experience. In the last decade, several point cloud coding solutions have been proposed using distinct approaches, notably two MPEG standards, addressing static and dynamic point cloud coding. More recently, learning-based coding approaches started to be considered also for point cloud coding. The performance of these solutions has been so competitive that JPEG already decided to develop a point cloud coding standard adopting this novel approach. This paper proposes the first learning-based rate control mechanism to minimize the complexity associated to the selection of appropriate coding parameters for the learning-based point cloud geometry codec adopted as the initial Verification Model for the development of the JPEG Pleno Learning-based Point Cloud Coding standard.

Deep learning (DL) based tools have recently reached performance levels similar to state-of-the-art hand-crafted methods for Point Cloud (PC) coding and classification. In 2022, JPEG issued a Call for Proposals for a Learning-based PC Coding (PCC) standard that envisions a unified representation, targeting both human visualization and computer vision tasks. This paper proposes the first DL-based Compressed Domain PC CLassifier (CD-PCCL), built on the PointGrid classifier, for geometry-only PCs coded with the current DL-based JPEG Pleno PCC Verification Model. The performance of compressed domain PC classification is studied against using voxel domain classification, notably for original, voxelized, and decompressed PCs. Experimental results with the ModelNet40 PC dataset show the proposed CD-PCCL can achieve significant PC classification gains regarding decompressed domain classification, while reducing the PC classifier complexity.

Recent advancements in Deep Learning (DL)-based architectures have demonstrated that integrating attention models can substantially enhance the performance across various tasks, including computer vision and visual coding. In accordance with this trend, this paper proposes a framework for incorporating attention models into DL-based Point Cloud (PC) geometry coding, namely JPEG Pleno PC Coding (JPEG PCC) and a definition of its key architectural design options. Experimental results show that the integration of attention models in JPEG PCC can provide a trade-off between compression and complexity, notably compression gains at the cost of an increase in model complexity and number of parameters. If the priority is on compression gains, the use of attention models can lead to a rate reduction up to 5.7%, for both the PSNR DI and PSNR D2 geometry quality metrics, at the cost of a 47% increase on the number of parameters and a 15.1% increase on the Floating-Point Operations (FLOPs) complexity. This obtained trade-off depends on the attention model integration configuration which can be defined depending on the application requirements.

Deep learning (DL)-based point cloud (PC) classification is a key computer vision task for many applications, notably autonomous driving, surveillance, and cultural heritage. In many application scenarios, PCs must be coded to reach practical rates for storage and transmission purposes, and thus they suffer from more or less intense compression artifacts. After the specification of two MPEG PC coding standards, DL-based PC coding has gained momentum, reaching competitive compression performance, especially for dense PCs. Since using decoded PCs, which may suffer from compression artifacts, may impact the final classification performance, the main goal of this paper is to study the impact of static PC geometry coding on DL-based classification. This study is performed on the ModelNet40 test dataset using the conventional G-PCC coding standard and the DL-based PC geometry codec which was the top performing solution responding to the recent JPEG Pleno PC Coding Call for Proposals. Two highly performing DL-based classifiers are used, considering the original PC geometry before and after voxelization, as well as the decoded PC geometry for different rates and qualities. As expected, coding has an impact on the classification performance, especially for the lower rates/qualities. For very sparse PCs, conventional coding still has advantage, contrarily to dense PCs, but this should change in the future with DL-based tools becoming the most natural solutions for both PC geometry coding and classification.

Point clouds have recently become a popular 3D representation model for many application domains, notably virtual and augmented reality. Since point cloud data is often very large, processing a point cloud may require that it be segmented into smaller clusters. For example, the input to deep learning-based methods like auto-encoders should be constant size point cloud clusters, which are ideally compact and non-overlapping. However, given the unorganized nature of point clouds, defining the specific data segments to code is not always trivial. This paper proposes a point cloud clustering algorithm which targets five main goals: i) clusters with a constant number of points; ii) compact clusters, i.e., with low dispersion; iii) non-overlapping clusters, i.e., not intersecting each other; iv) ability to scale with the number of points; and v) low complexity. After appropriate initialization, the proposed algorithm transfers points between neighboring clusters as a propagation wave, filling or emptying clusters until they achieve the same size. The proposed algorithm is unique since there is no other point cloud clustering method available in the literature offering the same clustering features for large point clouds at such low complexity.

Point clouds are gaining importance as the format to represent complex 3D objects and scenes, offering high user immersion and interaction, although at the cost of requiring massive data. Scalable coding is an important feature for point cloud coding, especially for real-time applications, where the fast and bitrate efficient access to a decoded point cloud is important; however, this issue is still rather unexplored in the literature. With the rise of deep learning methods as a promising solution for efficient coding, this paper proposes the first deep learning-based point cloud geometry scalable coding solution. Experimental results show that the proposed scalable coding solution consistently outperforms the MPEG standard for static point cloud geometry coding. In this way, a new research path is open for point cloud scalable coding technology.

Point clouds are a 3D visual representation format that has recently become fundamentally important for immersive and interactive multimedia applications. Considering the high number of points of practically relevant point clouds, and their increasing market demand, efficient point cloud coding has become a vital research topic. In addition, scalability is an important feature for point cloud coding, especially for real-time applications, where the fast and rate efficient access to a decoded point cloud is important; however, this issue is still rather unexplored in the literature. In this context, this paper proposes a novel deep learning-based point cloud geometry coding solution with resolution scalability via interlaced sub-sampling. As additional layers are decoded, the number of points in the reconstructed point cloud increases as well as the overall quality. Experimental results show that the proposed scalable point cloud geometry coding solution outperforms the recent MPEG Geometry-based Point Cloud Compression standard which is much less scalable.

Point clouds have recently become an important visual representation format, especially for virtual and augmented reality applications, thus making point cloud coding a very hot research topic. Deep learning-based coding methods have recently emerged in the field of image coding with increasing success. These coding solutions take advantage of the ability of convolutional neural networks to extract adaptive features from the images to create a latent representation that can be efficiently coded. In this context, this paper extends the deep-learning coding approach to point cloud coding using an autoencoder network design. Performance results are very promising, showing improvements over the Point Cloud Library codec often taken as benchmark, thus suggesting a significant margin of evolution for this new point cloud coding paradigm.

Point cloud rendering has a vital role in the user Quality of Experience for applications adopting point cloud based representations. While this is not a new area, it has recently become more relevant with the recent interest on point cloud coding by major standardization groups, notably JPEG and MPEG. The screened Poisson surface reconstruction is a state-of-the-art technique for generating a watertight surface mesh from the point cloud samples. While its screening component allows the surface to better fit the cloud points, this fitting may lead to undesired artifacts in the surface, notably when the point cloud is noisy. This paper proposes to improve this reconstruction method by making it more robust to noise by adopting a generalized Tikhonov regularization term. The proposed regularization approach smooths regions that should be flat while keeping the important details in the edges, thus creating more pleasant surface reconstructions.

Medical imaging technology has become essential for the improvement of medical practice. This led to advances in the technology, namely in image sampling resolutions, pixel bit-depth and inter slice resolution. Additionally, common use of medical images, the life expectancy of patients and legal restrictions led to increasing storage costs. Therefore, efficient compression of medical image data is in high demand, for archiving and transmission. In this work we propose to improve the compression efficiency of the Minimum Rate Predictors lossless encoder, by adding bi-directional prediction support and a histogram packing technique. The results show that the proposed method presents a higher compression efficiency than state-of-the-art HEVC encoder. The compression efficiency is improved by 20%, on average, when compared to HEVC and by 46.1% when compared with the original MRP algorithm.

Medical imaging compression is experiencing a growth in terms of usage and image resolution, namely in diagnostics systems that require a large set of images, like MRI or CT. Furthermore, legal and diagnosis restrictions impose the use of lossless compression and data archival for several years. These facts create a demand for more efficient compression tools, used for archiving and communication. In this work, we first evaluate the performance of traditional medical image compression algorithms against that of recent state of the art lossless image encoders. We then propose a method to improve the Minimum Rate Predictors lossless encoder, by exploiting inter picture redundancy in volumetric anatomical images. Results show that the proposed method is more efficient than state of the art encoders, such as HEVC, by about 28.8%, and achieves a gain of up to 57.8% in compression ratio when compared with traditional methods.