Light Fields (LFs) are an image modality derived from the plenoptic function, capable of representing scenes with a richer amount of visual information, making them well-suitable for immersive applications. To efficiently compress this data-intensive modality, the Joint Photographic Experts Group (JPEG) committee created the JPEG Pleno Part 2 standard with two profiles. This work focuses on the reference encoder and decoder implementations for the Baseline Block-Based Profile (BBBP), i.e., the JPEG PLeno Model (JPLM). Our main contribution lies in the proposal and analysis of a parallel implementation of JPLM codec using OpenMP, including the coding efficiency overhead of three solutions to support parallel decoding: two based on a marker defined in Part 2, called PNT; and another non-compliant version of the entropy codec. We show that it is possible to accelerate encoding up to$12\times $when using 24threads, and decoding up to$4\times $with 10 threads. In both cases, the memory overhead remained below 20% for the tested LFs. Although the parallel encoder produces bit-exact matches when compared to the sequential version, a PNT marker is required to ensure parallel decoding, which causes an increase of about 4% in terms of BD-Rate. On the other hand, we show that by updating the arithmetic codec to avoid the PNT, this overhead reduces to at most 0.22%.

Point cloud compression has been studied in standard bodies and we are here concerned with the Moving Picture Experts Group video-based point cloud compression (V-PCC) solution. Plenoptic point clouds (PPC) is a novel volumetric data representation wherein points are associated with colors in all viewing directions to improve realism. It is sampled as a number ($N_{c}$) of attribute colors per point. We propose a new method for the efficient video-based compression of PPC that is backwards compatible with the existing single-color V-PCC decoder. V-PCC generates three image atlases which are encoded using an image/video encoder. We assume there may be a reference color which is to be encoded as the main payload. We generate$N_{c}+3$atlases and we produce$N_{c}$differential images against the reference color image. Those difference images are pixel-wise transformed using an$N_{c}$-point discrete cosine transform, generating$N_{c}$transformed atlases which are encoded, forming the secondary payload. Such secondary information is the plenoptic enhancement to the point cloud. If there is no reference attribute, we skip the differences and use the lowest frequency of the transformed atlases as the main payload. Results are presented that show an unrivaled performance of the proposed method.

This paper proposes a rate-based candidate elimination strategy for Motion Estimation, which is considered one of the main sources of encoder complexity. We build from findings of previous works that show that selected motion vectors are generally near the predictor to propose a solution that uses the motion vector bitrate to constrain the candidate search to a subset of the original search window, resulting in less distortion computations. The proposed method is not tied to a particular search pattern, which makes it applicable to several ME strategies. The technique was tested in the VVC reference software implementation and showed complexity reductions of over 80% at the cost of an average 0.74% increase in BD-Rate with respect to the original TZ Search algorithm in the LDP configuration.

Recently finalized, the MPEG Video-based Point Cloud Compression (V-PCC) standard leverages existing video codecs to compress point clouds. This approach relies on existing video coding hardware accelerators to enable its fast adoption. However, the processing steps to project 3D point clouds into 2D frames still have a considerable complexity. Thus, we propose two modifications to TMC2, V-PCC’s test model, considering the memory access pattern: one reducing unnecessary memory allocation and the other increasing data locality through a set of pre-processing steps. Our approach achieved up to 46.31% encoding self-time reduction without changing the resulting bitstream. This paper also provides an analysis of our implementation that may help future V-PCC codecs achieve real-time encoding.

The Gaussian image filter is a compute-intensive approach to reduce undesirable artifacts and generally serves as a pre-processing technique for emerging applications related to visual computing systems. This work evaluates alternatives for the design of power-efficient Gaussian Filters. The proposed optimization strategy combines: 1) a refactored function to minimize the arithmetic operations, and 2) a design space exploration investigating different approximation scenarios applied to the full adders. The exact version of our refactored Gaussian Filter architecture reduces the total power consumption and the circuit area by 18% and 12%, respectively compared with the baseline Gaussian Filter architecture. Moreover, the combination of different approximation levels with the refactored architecture provides design options with power reductions from 21% to 59% compared with the baseline Gaussian Filter architecture.

The recently finalized Versatile Video Coding (VVC) standard brings a number of new tools to substantially improve coding efficiency, which resulted in a significant increase in complexity. Therefore, the use of such new standard in mobile devices requires not only the use of dedicated VLSI hardware architectures, but also the adoption of techniques that could reduce its complexity to meet the real-time and energy efficiency requirements. In this context, the most intensive encoding tools, such as the Fractional Motion Estimation (FME), are the first ones to be considered for optimization. Thereby, this work presents three different search patterns for the FME that are able to reduce the complexity of VVC by creating fixed search windows around the most relevant candidates. Our experiments show that these patterns result in acceptable decreases in coding efficiency, with average BD-Rates in the range of 0.56% - 0.33%, for the LD-P configuration, and 0.47% - 0.21% for the RA configuration, while allowing for significant hardware optimization. Comparing to a state-of-the-art FME hardware architecture that searches over 48 candidates the three proposed patterns can operate in higher frequencies to achieve the same throughput and require less area, leading to less power consumption and higher energy efficiency. Particularly, one of the three patterns may lead to a 60% area reduction and 53% less dynamic power consumption.

Video coding systems have to deal with a number of tradeoffs. The decision of adopting a specific distortion metric in the Fractional Motion Estimation (FME) step, for instance, presents a designer with a tradeoff between energy and coding efficiency. This paper analyzes such a tradeoff considering two of the most known and used distortion metrics, the Sum of Absolute Differences (SAD) and the Sum of Absolute Transformed Differences (SATD), within a High Efficiency Video Coding (HEVC)-compatible FME hardware architecture. We show that the SATD-based FME architecture is 1.94 times larger than the SAD-based one and consumes 2.07 times more energy.

Hybrid video standards rely on encoding prediction residues. To improve coding efficiency of inter-frame prediction, interpolated samples may be generated in fractional positions i.e., between neighbor pixels in the reference frame. However, performing Fractional Motion Estimation (FME) increases the overall encoder complexity. Since portable mobile devices are increasingly used to capture and reproduce videos, energy-efficient FME hardware accelerators are of utmost importance. In this work, we propose and evaluate a coding- and energy-efficient hardware design strategy for FME. Such strategy addresses the main weaknesses of the architectures found in the literature. The architecture designed as case study can achieve 2160p@120fps for the HEVC 8×8 FME. We also provide an insightful area and power breakdown of the synthesized design, to drive the design of FME hardware towards further energy improvements.

The efficiency improvements achieved by new video coding standards come at the cost of a huge increase in the encoder computational complexity. Paradoxically, such increasing complexity is commonly addressed by methods that have an adverse effect on coding efficiency. In this work, we propose a method to reduce the complexity of HEVC Hadamard ME, without compromising coding efficiency. Our method relies on a new metric named Absolute First Difference (AFD), which is able to eliminate impossible candidates using only a few operations. Despite its simplicity, AFD filters an average of 13.52% of SATDs in the HEVC reference model (HM), without reducing coding efficiency. Moreover, our method is able to filter up to 37.9% of SATDs for video conferencing and up to 64% for screen content.

State-of-the-art video coding standards adopt large block and transform sizes. Moreover, as resolutions keep growing, there is a trend in adopting even larger structures in future video encoders, resulting in higher complexity. Therefore, the design of energy-efficient architectures for variable block size distortion metrics are key to keep the energy requirements of battery devices under a reasonable budget. The Hadamard-based Sum of Absolute Transformed Differences (SATD) is used as distortion metric in several steps of encoding, increasing the overall encoding efficiency at the cost of rising complexity. In this work, we propose two main approaches for SATD calculation of N × N block sizes. One using a Transpose Buffer (TB) and another one using a Linear Buffer (LB). Furthermore, we synthesized four sizes (4 × 4 up to 32 × 32) of each main SATD architecture to evaluate their area and energy estimates. The results show a large increase in area for TB-SATD, while LB-SATD area increases in a much smaller pace. On the other hand, the TB-SATD synthesized for Low-Vdd/High-Vt show up to be the most energy-efficient architectures for all sizes.

Although pel decimation has been widely used to reduce the computational effort in video coding, there is no consensus about the optimal subsampling pattern. This paper presents an extensive analytical and statistical comparison of several different subsampling patterns using analysis of variance. The investigation includes commonly used patterns as well as seven proposed ones. The experiments were conducted on 19 video samples in a range of five resolutions (being nine videos at 1080p) and 30 bitrates by using the state-of-the-art x264 encoder running successive elimination exhaustive search. Two objective quality metrics were reported, PSNR and DSSIM, resulting in 10680 experimental points. The analysis of such amount of data allowed us to conclude that the proposed 4:3 ratio shows less than 5% in DSSIM and 1% in PSNR losses, being more than two times faster than full sampling. Compared with higher decimation, it presents a better trade-off between speedup and quality loss.