The need for more realistic 3D scene representations has fomented the development of models for a wide range of applications. In this context, solutions that attempt to model the light's behavior through the plenoptic function have provided considerable advancements using neural-based approaches, often presenting a trade-off between rendering time and model sizes. In this work, we propose a pruning framework to reduce the sizes of these models by computing the visibility over the training data, applicable to different 3D scene representations. In particular, we implement first a solution suitable for the 3D Gaussian Splatting, and then we exemplify the solution for the Neural Radiance Fields (NeRF)-style of rendering using PlenOctrees. We show that our pruning solution produces smaller models in terms of the number of elements – be they voxels, points, or Gaussians – with minimal losses in terms of rendering novel views. We further assess our solution by combining it with state-of-the-art (SOTA) compression solutions for both rendering schemes. Results over the NeRF-Synthetic dataset show comparable metrics to the SOTA for PlenOctrees, achieving marginal gains for lower bitrates. For 3DGS, the combination of our pruning method and compression solutions achieves a compression ratio of up to 37.5 times over the uncompressed 3DGS models, with only a 0.5 dB decrease in rendering quality. When compared against other SOTA compression methods, our solution produces models 1.4 times smaller, with less than a 0.1 dB loss over novel views for synthetic data, and models 1.9 times smaller with less than 0.2 dB loss when synthesizing novel views on real-world, outdoor content.

3D scene representation has been a central theme of study for a wide range of applications, and the representation of light behavior is one of the relevant topics when producing realistic models. In this work, we create a framework to assess the representation of non-Lambertian scenes by generating a pipeline to create plenoptic point clouds (PPCs) systematically and evaluating them against implicit solutions, such as Neural Radiance Fields (NeRF)-like models. We compare such approaches according to rendering quality and compression efficiency. On the compression side, we propose an encoding scheme for PPC, leveraging the occlusion masks of the points and the Moving Picture Expert Group's (MPEG) Geometry-Based Solid Content Test Model (GeS-TM). Rendering results over the training views show that the uncompressed PPC outperforms 3D Gaussian Splatting (3DGS) by 1.51 dB, on average, for the 8 scenes of the NeRF Synthetic 360 dataset. In compression efficiency, 3DGS outperforms the compressed PPCs by 0.7 dB in BD-PSNR on average. Our occlusion-aware encoding scheme reduces the size of uncompressed PPCs up to 800 times, outperforming current encoding schemes for PPC by 1.9 dB in BD-PSNR.

The volumetric representation of human interactions is one of the fundamental domains in the development of immersive media productions and telecommunication applications. Particularly in the context of the rapid advancement of Extended Reality (XR) applications, this volumetric data has proven to be an essential technology for future XR elaboration. In this work, we present a new multimodal database to help advance the development of immersive technologies. Our proposed database provides ethically compliant and diverse volumetric data, in particular 27 participants displaying posed facial expressions and subtle body movements while speaking, plus 11 participants wearing head-mounted displays (HMDs). The recording system consists of a volumetric capture (VoCap) studio, including 31 synchronized modules with 62 RGB cameras and 31 depth cameras. In addition to textured meshes, point clouds, and multi-view RGB-D data, we use one Lytro Illum camera for providing light field (LF) data simultaneously. Finally, we also provide an evaluation of our dataset employment with regard to the tasks of facial expression classification, HMDs removal, and point cloud reconstruction. The dataset can be helpful in the evaluation and performance testing of various XR algorithms, including but not limited to facial expression recognition and reconstruction, facial reenactment, and volumetric video. HEADSET and its all associated raw data and license agreement will be publicly available for research purposes.

Plenoptic point clouds (PPC) are novel data structures that represent the light from different viewing directions in order to provide a higher degree of realism to regular point clouds. This is achieved by associating each point to multiple colors instead of a single one. Here, we present a method to efficiently compress the attributes of a PPC, consisting of a Karhunen-Loève transform over the color attributes followed by multiple attribute coders with intra prediction capability. This compression scheme can be incorporated within the MPEG's geometry-based PCC (G-PCC) standard, using any of G-PCC's existing solutions for attribute coding. Compression performance assessment using PPCs of different spatial resolutions reveals competitive results in comparison to existing methods, such as RAHT-based or video-based PCC solutions. We believe our coder to be the new state of the art.

Point cloud compression has been studied in standard bodies and we are here concerned with the Moving Picture Experts Group video-based point cloud compression (V-PCC) solution. Plenoptic point clouds (PPC) is a novel volumetric data representation wherein points are associated with colors in all viewing directions to improve realism. It is sampled as a number ($N_{c}$) of attribute colors per point. We propose a new method for the efficient video-based compression of PPC that is backwards compatible with the existing single-color V-PCC decoder. V-PCC generates three image atlases which are encoded using an image/video encoder. We assume there may be a reference color which is to be encoded as the main payload. We generate$N_{c}+3$atlases and we produce$N_{c}$differential images against the reference color image. Those difference images are pixel-wise transformed using an$N_{c}$-point discrete cosine transform, generating$N_{c}$transformed atlases which are encoded, forming the secondary payload. Such secondary information is the plenoptic enhancement to the point cloud. If there is no reference attribute, we skip the differences and use the lowest frequency of the transformed atlases as the main payload. Results are presented that show an unrivaled performance of the proposed method.

Recently finalized, the MPEG Video-based Point Cloud Compression (V-PCC) standard leverages existing video codecs to compress point clouds. This approach relies on existing video coding hardware accelerators to enable its fast adoption. However, the processing steps to project 3D point clouds into 2D frames still have a considerable complexity. Thus, we propose two modifications to TMC2, V-PCC’s test model, considering the memory access pattern: one reducing unnecessary memory allocation and the other increasing data locality through a set of pre-processing steps. Our approach achieved up to 46.31% encoding self-time reduction without changing the resulting bitstream. This paper also provides an analysis of our implementation that may help future V-PCC codecs achieve real-time encoding.

This paper proposes a lossy intra-frame coder of the geometry information of voxelized point clouds. Using an alternative approach to the widespread octree representation, this method represents the point cloud as an array of binary images. This algorithm works recursively using a dyadic decomposition that splits an interval of slices in two smaller intervals, depicting a binary tree traversal, and transmitting the occupancy information of each interval. The sequence of bi-level images are encoded in a lossless fashion until a fixed point in the tree, from where the algorithm “skips” the dyadic slicing and transmits all the k remaining slices as leaves of the tree, which are then encoded in a lossy fashion. The performance assessment shows that the proposed method outperforms state-of-the-art intra coders of lossy geometry for medium to higher bitrates on the public point cloud datasets tested.