Wavelet transforms have been a relevant topic in signal processing for many years. One of the most common strategies when designing wavelet transforms is the use of lifting schemes, known for their perfect reconstruction properties and flexible design. This paper introduces a novel 2D non-separable lifting design methodology based on deep learning architectures. The proposed method is assessed within the context of end-to-end lossless image compression. The attained results highlight a significant 99% reduction of the number of learnable coefficients, with a direct impact on the computational complexity, while still achieving higher compression performance when compared to other end-to-end state-of-the-art wavelet-based frameworks. The code for the models used to obtain the reported results are available at https://github.com/joaoparracho/2D-NSWT-LCLS.

Common representations of light fields use four-dimensional data structures, where a given pixel is closely related not only to its spatial neighbours within the same view, but also to its angular neighbours, co-located in adjacent views. Such structure presents increased redundancy between pixels, when compared with regular single-view images. Then, these redundancies are exploited to obtain compressed representations of the light field, using prediction algorithms specifically tailored to estimate pixel values based on both spatial and angular references. This paper proposes new encoding schemes which take advantage of the four-dimensional light field data structures to improve the coding performance of Minimum Rate Predictors. The proposed methods expand previous research on lossless coding beyond the current state-of-the-art. The experimental results, obtained using both traditional datasets and others more challenging, show bit-rate savings no smaller than 10%, when compared with existing methods for lossless light field compression.

A common approach used in omnidirectional video coding is based on frame splitting into tiles, allowing partial delivery of only the subset of tiles that is necessary to render the user’s current viewing region, defined as a specific viewport or Field-of-View (FoV). Since tiles can be independently encoded, such mechanism provides a flexible solution for encoding planar representations with ultra-high definition (UHD), such as the Equirectangular Projection (ERP), using Versatile Video Coding (VVC). By only selecting and transmitting the coded data that is required to render the necessary FoV, rather than the full 360°, a great deal of bandwidth can be saved. While current solutions are based on splitting the omnidirectional video frames into tiles of equal size, this paper proposes a new approach based on adaptive tile size, driven by visual attention. Those regions where the visual attention is higher are partitioned in smaller tiles to obtain higher bit rate granularity, allowing to decode the most frequent FoVs with minimum out-of-FoV pixels and reduced bandwidth. Optimal tile boundaries are found by solving a lagrangian minimisation problem with a cost function that achieves the best tradeoff between the standard deviation and the average attention-weighted bit rate per tile. The experimental results show that an average of 7.17% and 17.73% of bit rate savings is obtained in comparison with conventional tilling methods for the commonly used FoVs of $90^{\circ} \times 90^{\circ}$ and $45^{\circ} \times 45^{\circ}$, respectively.

Encoding 360° video with ultra high spatial resolution requires high bitrates to guarantee acceptable QoE in video delivery services. However, since in general the full Field-of-View (FoV), i.e., 360°, is not required at once, a great deal of bandwidth can be saved if only a limited FoV is delivered, according to content relevancy or/and user demand. This work addresses this problem using the concept of Adaptive Resolution Change (ARC) defined in the forthcoming Versatile Video Coding (VVC) standard, by dynamically mapping the full FoV into multiple video frames with different spatial resolutions. Those FoVs attracting more visual attention are encoded with higher resolution while the others are encoded with lower resolution, thus without compromising the visual quality and resolution of the most relevant regions. The simulation results show that the proposed adaptive coding scheme is able to deliver high quality video for the most relevant FoV at any time instant, achieving a maximum bitrate reduction of 37.2%.

This paper describes a robust depth estimation algorithm for multi-focus plenoptic images. The main feature of the proposed method consists of a hybrid template matching scheme built-upon intensity and local phase information, which adapts to the blurriness of neighbouring lenslet microimages. By reducing the impact of defocusblur on the template matching accuracy, the proposed method efficiently handles the varying triangulation baseline over the depth-of-field, thus discarding the need for scene-related information such as the expected range of disparities. Experimental results demonstrate the robustness of the proposed method over the most used commercially available depth estimation algorithm, achieving a reduction of 73% on the depth estimation error.

Efficient light field encoders take advantage of the inherent 4D data structures to achieve high compression performance. This is accomplished by exploiting the redundancy of co-located pixels in different sub-aperture images (SAIs) through prediction and/or transform schemes to find a m ore compact representation of the signal. However, in image regions with higher disparity between SAIs, such scheme's performance tends to decrease, thus reducing the compression efficiency. This paper introduces a reversible pre-processing algorithm for disparity compensation that operates on the SAI domain of light field data. The proposed method contributes to improve the transform efficiency of the encoder, since the disparity-compensated data presents higher correlation between co-located image blocks. The experimental results show significant improvements in the compression performance of 4D light fields, achieving Bjontegaard delta rate gains of about 44% on average for MuLE codec using the 4D discrete cosine transform, when encoding High Density Camera Arrays (HDCA) light field images.

This letter proposes a method to obtain integer reversible discrete cosine transforms for generic transform-based coding schemes. The novelty of the proposed method, which is based on decomposition of the DCT-II matrix into two triangular and one diagonal matrices, is twofold: (i) the new matrices can be of arbitrary size, i.e., any square N × N dimension, thus suitable for applications where non power-of-2 dimensions are required; (ii) they can be designed with adjustable precision in a trade-off with the number of representation bits. Furthermore, improvements are also proposed over the base scheme to avoid numerical issues when working with large matrices and to obtain more reliable approximations. The performance evaluation demonstrate the effectiveness of the proposed transforms to approximate the coding gain capabilities of the original DCT-II.

This paper presents a method to improve the lossless compression efficiency of light field encoding based on Minimum Rate Predictors (MRP). The proposed method relies on the use of multiple references, either micro-images or sub-aperture images, to provide a richer set of correlated pixels for prediction. The results show better compression ratios than conventional versions of MRP, for both representation formats (micro-image and sub-aperture image arrays), achieving gains ranging from 16.9% to 21.2%. Furthermore, it is also shown that the proposed method consistently outperforms the state-of-the-art lossless encoders HEVC and JPEG-LS.

High quality omnidirectional video requires ultra high resolution formats encoded with very high bit rates to guarantee acceptable QoE in video delivery services. Since in general the full FoV, i.e., 360°, is not required at once by users, rather than agnostic encoding of the whole 360° video, this work proposes a flexible coding approach where the full FoV is mapped into video frames and efficiently encoded using intra-FoV prediction. By avoiding inter-FoV prediction, the proposed approach enables independent decoding of one or more FoVs extracted from a single compressed stream containing the full FoV video. To achieve improved quality in those FoVs which attract more visual attention, non-uniform coding is proposed for the new Versatile Video Coding standard (VVC), using perceptually-driven quantisation for each FoV. This strategy, makes use of visual attention maps to decrease the overall bit rate without compromising the quality of the most relevant regions. The simulation results show that the proposed coding mechanism achieves consistent quality gains in the relevant FoV without significant losses in the remaining ones. In comparison with the reference VVC, the proposed method is able to achieve average quality gains up to 1.56 dB and to efficiently adapt the coding parameters to the visual attention information.

This paper presents a method to improve disparity maps computed from light field images using structure tensor methods, which tend to expand the borders of the occluding objects, enlarging their silhouette. The proposed method relies on the fact that such regions of the silhouette are defined by a mismatch between the disparity map edges, computed by the structure tensor, and those of the corresponding epipolar plane images (EPI) representing the light field. The proposed silhouette improvement method determines a correspondence between EPI edges and the disparity map edges, identifying the erroneous silhouette regions. The disparity map is corrected by using neighbouring values computed with high structure tensor reliability. The achieved results show that the disparity map is improved both around object edges and overall, reducing the MSE by 47.9% in comparison with other methods also based on the structure tensor.

The performance of focus metrics in the evaluation of refocused images from light fields is investigated in this paper. To this aim, the paper presents a comprehensive study on the performance of a large set of different focus metrics (34 in total) in the evaluation of patch-based extended depth-of-field reconstructed images (e.g., all-in-focus). The new findings of this work demonstrate that optimal reconstruction of extended depth-of-field images, from light fields captured with focused plenoptic cameras, is not consistent with the focus level figures computed by available focus metrics. The results show that a higher focus level, as given by such computational methods, does not actually correspond to better all-in-focus images, thus indicating that currently available focus metrics are not suitable for evaluating the quality of extended depth-of-field reconstruction. The results obtained through subjective evaluation confirm that objective focus metrics fail to indicate the best focused images. Finally, the paper presents recommendations to adapt existing metrics to extended depth-of-field reconstruction in plenoptic imaging.