Focused plenoptic cameras can record spatial and angular information of the light field (LF) simultaneously with higher spatial resolution relative to traditional plenoptic cameras, which facilitate various applications in computer vision. However, the existing plenoptic image compression methods present ineffectiveness to the captured images due to the complex micro-textures generated by the microlens relay imaging and long-distance correlations among the microimages. In this article, a lossy end-to-end learning architecture is proposed to compress the focused plenoptic images efficiently. First, a data preprocessing scheme is designed according to the imaging principle to remove the sub-aperture image ineffective pixels in the recorded light field and align the microimages to the rectangular grid. Then, the global attention module with large receptive field is proposed to capture the global correlation among the feature maps using pixel-wise vector attention computed in the resampling process. Also, a new image dataset consisting of 1910 focused plenoptic images with content and depth diversity is built to benefit training and testing. Extensive experimental evaluations demonstrate the effectiveness of the proposed approach. It outperforms intra coding of HEVC and VVC by an average of 62.57% and 51.67% bitrate reduction on the 20 preprocessed focused plenoptic images, respectively. Also, it achieves 18.73% bitrate saving and generates perceptually pleasant reconstructions compared to the state-of-the-art end-to-end image compression methods, which benefits the applications of focused plenoptic cameras greatly. The dataset and code are publicly available athttps://github.com/VincentChandelier/GACN.

Learned image compression has exhibited promising compression performance, but variable bitrates over a wide range remain a challenge. State-of-the-art variable rate methods compromise the loss of model performance and require numerous additional parameters. In this paper, we present a Quantization-error-aware Variable Rate Framework (QVRF) that utilizes a univariate quantization regulator a to achieve wide-range variable rates within a single model. Specifically, QVRF defines a quantization regulator vector coupled with predefined Lagrange multipliers to control quantization error of all latent representation for discrete variable rates. Additionally, a reparameterization method makes QVRF compatible with round quantizer and integer entropy coding. Exhaustive experiments demonstrate that existing fixed-rate VAE-based methods equipped with QVRF can achieve wide-range continuous variable rates within a single model without significant performance degradation. Furthermore, QVRF outperforms contemporary variable-rate methods in rate-distortion performance with minimal additional parameters. The code is available at https://github.com/bytedance/QRAF.

The plenoptic 2.0 video can record a time-varying dense light field, which benefits many immersive visual applications such as AR/VR. However, traditional inter motion estimation methods perform inefficiently in such kinds of video sequences due to the distinctive temporal characteristics caused by the imaging principle. In this paper, a microimage-based two- step search (MTSS) is proposed to achieve a better trade-off between coding performance and coding complexity. Based on microimage focus variation analysis in imaging dynamic scenes, a microlens-diameter and matching-distance spatial search with local refinement is proposed to exploit the image correlations among the microimage and to compensate the defocused inaccuracy. Implementing the proposed motion estimation in H.266 platform VTM-11.0 and comparing with the state-of-the-art methods, obvious compression efficiency improvements are achieved with limited complexity increment, which benefits the standardization of plenoptic video coding.

Light field image becomes one of the most promising media types for immersive video applications. In this paper, we propose a novel end-to-end spatial-angular-decorrelated network (SADN) for high-efficiency light field image compression. Different from the existing methods that exploit either spatial or angular consistency in the light field image, SADN decouples the angular and spatial information by dilation convolution and stride convolution in spatial-angular interaction, and performs feature fusion to compress spatial and angular information jointly. To train a stable and robust algorithm, a large-scale dataset consisting of 7549 light field images is proposed and built. The proposed method provides 2.137 times and 2.849 times higher compression efficiency relative to H.266/VVC and H.265/HEVC inter coding, respectively. It also outperforms the end-to-end image compression networks by an average of 79.6% bitrate saving with much higher subjective quality and light field consistency.

In video coding, the illumination information of video scenes is usually hard to be compressed due to the complex and unpredictable illumination variations. To simplify the problem, many prediction algorithms assume a linear correlation of illumination variations existing between frames by constructing corresponding linear model (LM), such as Weighted Prediction (WP) method. The assumption is suitable for uniform illumination variations with lager area, but ineffective in sharp illumination variations in small area. In this paper, we propose a piecewise linear model based local illumination compensation (PLMLIC) approach to further compensate sharp illumination variations in small area. When PLMLIC is applied to a coding unit (CU), we first use multiple reference lines, i.e. allow not only the nearest reference line but also long-distance reference lines to be the neighbouring samples of the current CU. Then the neighbouring samples and their corresponding reference samples are classified into 2 groups, based on which PLMLIC parameters are derived for each group. Finally, the current CU is predicted by the corresponding PLMLIC parameters. Experimental results show that 0.23% BD-rate savings on average can be achieved for lowdelay configuration based on Enhanced Compression Model (ECM) beyond VVC.

Plenoptic 2.0 videos that record time-varying light fields by focused plenoptic cameras are prospective to immersive visual applications due to capturing dense sampled light fields with high spatial resolution in the rendered sub-apertures. In this paper, an intra prediction method is proposed for compressing multi-focus plenoptic 2.0 videos efficiently. Based on the estimation of zooming factor, novel gradient-feature-based zooming, adaptive-bilinear-interpolation-based tailoring and inverse-gradient-based boundary filtering are proposed and executed sequentially to generate accurate prediction candidates for weighted prediction working with adaptive skipping strategy. Experimental results demonstrate the superior performance of the proposed method relative to HEVC and state-of-the-art methods.

Recently, plenoptic image has attracted great attentions because of its applications in various scenarios. However, high resolution and special pixel distribution structure bring huge challenges to its storage and transmission. In order to adapt compression to the structural characteristic of plenoptic image, in this paper, we propose a Data Structure Adaptive 3D-convolutional(DSA-3D) autoencoder. The DSA-3D autoencoder enables up-sampling and down-samping the sub-aperture sequence along the angular resolution or spatial resolution, thereby avoiding the artifacts caused by directly compressing plenoptic image and achieving better compression efficiency. In addition, we propose a special and efficient Square rearrangement to generate sub-aperture sequence. We compare Square with Zigzag sub-aperture sequence rearrangements, and analyzed the compression efficiency of block image compression and whole image compression. Compared with traditional hybrid encoders HEVC, JPEG2000 and JPEG PLENO(WaSP), the proposed DSA-3D(Square) autoencoder achieves a superior performance in terms of PSNR metrics.