The Joint Photographic Experts Group (JPEG) is currently addressing the challenges in assessing plenoptic image quality by developing new standards for both subjective and objective assessment. This process entails revisiting existing recommendations and establishing a visual quality assessment (QA) framework that considers the unique aspects of plenoptic data. The focus is currently on the light field modality, with efforts to gather expert contributions and develop tools that cater both subjective and objective QA, ensuring that the evolving requirements of plenoptic imaging QA are met. This paper presents the JPEG Pleno subjective test tool, a pivotal element in JPEG’s standardization endeavor, designed to facilitate a range of subjective QA experiments that steer the decisions during a standardization process.

In this work we study the 4D sparsity of light fields using as main tool the 4D- Discrete Cosine Transform. We analyze the two JPEG Pleno light field datasets, namely the lenslet-based and the High-Density Camera Array (HDCA) datasets. The results suggest that the lenslets datasets exhibit a high 4D redundancy, with a larger inter-view sparsity than the intra-view one. For the HDCA datasets, there is also 4D redundancy worthy to be exploited, yet in a smaller degree. Unlike the lenslets case, the intra-view redundancy is much larger than the inter-view one. The results and conclusions of this first study for light field imaging may have a strong impact on the current and future design of efficient codecs for this type of emerging data, notably in the context of the JPEG Pleno standard.

Light fields aim to represent visual information in 3D space. They are 4D structures that contain the images of a given scene from a sampled 2D range of viewpoints. When acquired using a lenslet camera, in addition to the ordinary intra-view redundancy, these views have a great deal of inter-view redundancy. In this work we propose a light field codec that fully exploits the 4D redundancy of light fields by using a 4D transform and hexadeca-trees. It initially divides the light field into 4D blocks and computes a 4D Discrete Cosine Transform of each one. Then the transform coefficients of the 4D block are grouped using hexadeca-trees on a bitplane-by-bitplane basis, and the generated stream is encoded using an adaptive arithmetic coder. The proposed codec has been employed to encode the JPEG Pleno lenslet light fields. The rate-distortion results have been assessed using test conditions comparable to the ones presented at the ICIP 2017 Light Field Coding Grand Challenge. The proposed codec, despite being conceptually simple, achieves competitive rate-distortion performance.

The fusion of visible-light and infrared videos has applications in several areas, and is an active research topic. To this end, it is common to employ complex fusion methods that take into account spatial and/or temporal information in the videos. In this paper we propose a video fusion method that is based on a motion-compensated, two-band temporal sub-band decomposition. The alignment provided by the motion vectors, besides providing a reduction in the registration errors of the input images, allows the use of a simple fusion rule to the temporal sub-bands. The results indicate that the use of the proposed exploration of the temporal information alone is quite effective, and gives objective fusion quality results that compare favorably to more sophisticated methods based on complete spatiotemporal information.

The representation of data in terms of its statistical properties is valuable in many applications. This work uses statistics obtained from 4D scene geometry to characterize, in terms of redundancy, the content produced by lenslet-based light field cameras and by high-density arrays of cameras for the JPEG Pleno Call for Proposals on Light Field Coding. This paper proposes a novel so-called geometric space-view redundancy (GSVR) descriptor, which is able to characterize the amount of redundancy in light fields thus bringing information about the trade-offs involved in effectively exploring redundancy for efficient coding. The redundancy is here measured by the probability, for each block size and range of views, that the image of a given 3D point belongs to the block in all views. For a given probability, the GSVR descriptor models the spaceview correlation, i.e. the correlation between the intra-view block dimensions and the number of views. Therefore, it is a descriptor that has application on dataset selection and encoder control and optimization. The JPEG Pleno datasets are analyzed in terms of the GSVR descriptor in all views.

Directional intra prediction plays an important role in current state-of-the-art video coding standards. In directional prediction, neighbouring samples are projected along a specific direction to predict a block of samples. Ultimately, each prediction mode can be regarded as a set of very simple linear predictors, a different one for each pixel of a block. Therefore, a natural question that arises is whether one could use the theory of linear prediction in order to generate intra prediction modes that provide increased coding efficiency. However, such an interpretation of each directional mode as a set of linear predictors is too poor to provide useful insights for their design. In this paper, we introduce an interpretation of directional prediction as a particular case of linear prediction, which uses the first-order linear filters and a set of geometric transformations. This interpretation motivated the proposal of a generalized intra prediction framework, whereby the first-order linear filters are replaced by adaptive linear filters with sparsity constraints. In this context, we investigate the use of efficient sparse linear models, adaptively estimated for each block through the use of different algorithms, such as matching pursuit, least angle regression, least absolute shrinkage and selection operator, or elastic net. The proposed intra prediction framework was implemented and evaluated within the state-of-the-art high efficiency video coding standard. Experiments demonstrated the advantage of this predictive solution, mainly in the presence of images with complex features and textured areas, achieving higher average bitrate savings than other related sparse representation methods proposed in the literature.

This paper presents a new intra prediction method for efficient image coding, based on linear prediction and sparse representation concepts, denominated sparse least-squares prediction (SLSP). The proposed method uses a low order linear approximation model which may be built inside a predefined large causal region. The high flexibility of the SLSP filter context allows the inclusion of more significant image features into the model for better prediction results. Experiments using an implementation of the proposed method in the state-of-the-art H.265/HEVC algorithm have shown that SLSP is able to improve the coding performance, specially in the presence of complex textures, achieving higher coding gains than other existing intra linear prediction methods.

Stereoscopic video systems aim to confer a true three-dimensional (3D) view of the real world. However, in the process of 3D image acquisition and display, eventual distortions may affect the 3D perceptual quality, leading to visual discomfort. As visual comfort is a key issue when dealing with 3D acceptance, subjective tests (that are time consuming and cannot provide quality scores on the fly) have been conducted in order to establish 3D assessment factors. In this work, we use the ground truth of a publicly available database in order to develop a method to automatically assess the visual comfort of stereoscopic imagery. The assessment has the objective of assisting in the adjustment of stereo camera baselines. The method employs stereo vision geometry and stereo matching algorithms to estimate depth planes, for each video frame, in order to associate their locations with different visual comfort zones. The proposed method shows a high level of correlation with subjective tests, comparing favorably with another proposal from the literature.

A complete encoding solution for efficient intra-based depth map compression is proposed in this paper. The algorithm, denominated predictive depth coding (PDC), was specifically developed to efficiently represent the characteristics of depth maps, mostly composed by smooth areas delimited by sharp edges. At its core, PDC involves a directional intra prediction framework and a straightforward residue coding method, combined with an optimized flexible block partitioning scheme. In order to improve the algorithm in the presence of depth edges that cannot be efficiently predicted by the directional modes, a constrained depth modeling mode, based on explicit edge representation, was developed. For residue coding, a simple and low complexity approach was investigated, using constant and linear residue modeling, depending on the prediction mode. The performance of the proposed intra depth map coding approach was evaluated based on the quality of the synthesized views using the encoded depth maps and original texture views. The experimental tests based on all intra configuration demonstrated the superior rate-distortion performance of PDC, with average bitrate savings of 6%, when compared with the current state-of-the-art intra depth map coding solution present in the 3D extension of a high-efficiency video coding (3D-HEVC) standard. By using view synthesis optimization in both PDC and 3D-HEVC encoders, the average bitrate savings increase to 14.3%. This suggests that the proposed method, without using transform-based residue coding, is an efficient alternative to the current 3D-HEVC algorithm for intra depth map coding.

The accurate geometric calibration of thermal infrared (IR) cameras is of vital importance in many computer vision applications. In general, the calibration procedure consists of localizing a set of calibration points within a calibration image. This set is subsequently used to solve for the camera parameters. However, due to the physical limitations of the IR acquisition process, the localization of the calibration points often poses difficulties, subsequently leading to unsatisfying calibration results. In this work a novel IR camera calibration approach is introduced. It is able to localize the calibration points within the images of a conventional calibration board consisting of miniature light-bulbs with improved accuracy. Our algorithm models the radiation pattern of each light bulb as an ellipse and considers the center of mass of the extracted ellipsoidal region as the starting calibration point, which is refined iteratively using alternating mappings to and from an undistorted grid model. The proposed processing chain leads to a significantly reduced calibration error when compared to the state-of-the-art. Furthermore, the proposed methodology can also be used to calibrate visible-light cameras thus being suitable for the calibration of multiple camera rigs involving both visible-light and IR cameras.

This paper presents a novel approach to compress depth maps envisioned for virtual view synthesis. This proposal uses a sophisticated prediction model, combining the HEVC intra prediction modes with a flexible partitioning scheme. It exhaustively evaluates the prediction modes for a large amount of block sizes, in order to find the minimum coding cost for each depth map block. Unlike HEVC, no transform is used, the residue being trivially encoded through the transmission of just its mean value. The experimental results show that, when the encoding evaluation metric is the quality of the view synthesized using the encoded depth map against the map encoding rate, the proposed algorithm generates reconstructed depth maps that provide, for most bitrates, some of the best performances among state-of-the-art depth maps encoders. In addition, it runs approximately as fast as the HEVC HM.

Multiscale transforms are among the most popular techniques in the field of pixel-level image fusion. However, the fusion performance of these methods often deteriorates for images derived from different sensor modalities. In this paper, we demonstrate that for such images, results can be improved using a novel undecimated wavelet transform (UWT)-based fusion scheme, which splits the image decomposition process into two successive filtering operations using spectral factorization of the analysis filters. The actual fusion takes place after convolution with the first filter pair. Its significantly smaller support size leads to the minimization of the unwanted spreading of coefficient values around overlapping image singularities. This usually complicates the feature selection process and may lead to the introduction of reconstruction errors in the fused image. Moreover, we will show that the nonsubsampled nature of the UWT allows the design of nonorthogonal filter banks, which are more robust to artifacts introduced during fusion, additionally improving the obtained results. The combination of these techniques leads to a fusion framework, which provides clear advantages over traditional multiscale fusion approaches, independent of the underlying fusion rule, and reduces unwanted side effects such as ringing artifacts in the fused reconstruction.

In this work we propose a fusion framework based on undecimated wavelet transforms with spectral factorization which includes information about the presence of targets within the infrared (IR) image to the fusion process. For this purpose a novel IR segmentation algorithm that extracts targets from low-contrast environments and suppresses the introduction of spurious segmentation results is introduced. Thereby, we ensure that the most relevant information from the IR image is included in the fused image, leading to a more accurate representation of the captured scene. Moreover, we propose the use of a novel, hybrid fusion scheme which combines both pixeland region-level information to guide the fusion process. This turns the fusion process more robust against possible segmentation errors which represents a common source of problems in region-level image fusion. The combination of these techniques leads to a novel fusion framework which is able to improve the fusion results of its pure pixel-level counterpart without target extraction. Additionally, traditional pixel-level fusion approaches, based on state-of-the-art transforms such as the Nonsubsampled Contourlet Transform and the Dual-Tree ComplexWavelet Transform, are significantly outperformed by the use of the proposed set of methods.

The importance to develop more efficient 3D and multiview data representation algorithms results from the recent market growth for 3D video equipments and associated services. One of the most investigated formats is video+depth which uses depth image based rendering (DIBR) to combine the information of texture and depth, in order to create an arbitrary number of views in the decoder. Such approach requires that depth information must be accurately encoded. However, methods usually employed to encode texture do not seem to be suitable for depth map coding.

Infrared focal-plane array (IRFPA) detectors suffer from fixed-pattern noise (FPN) that degrades image quality, which is also known as spatial nonuniformity. FPN is still a serious problem, despite recent advances in IRFPA technology. This paper proposes new scene-based correction algorithms for continuous compensation of bias and gain nonuniformity in FPA sensors. The proposed schemes use recursive least-square and affine projection techniques that jointly compensate for both the bias and gain of each image pixel, presenting rapid convergence and robustness to noise. The synthetic and real IRFPA videos experimentally show that the proposed solutions are competitive with the state-of-the-art in FPN reduction, by presenting recovered images with higher fidelity.

In this article we propose to compress depth maps using a coding scheme based on multiscale recurrent pattern matching and evaluate its impact on depth image based rendering (DIBR). Depth maps are usually converted into gray scale images and compressed like a conventional luminance signal. However, using traditional transform-based encoders to compress depth maps may result in undesired artifacts at sharp edges due to the quantization of high frequency coefficients. The Multidimensional Multiscale Parser (MMP) is a pattern matching-based encoder, that is able to preserve and efficiently encode high frequency patterns, such as edge information. This ability is critical for encoding depth map images. Experimental results for encoding depth maps show that MMP is much more efficient in a rate-distortion sense than standard image compression techniques such as JPEG2000 or H.264/AVC. In addition, the depth maps compressed with MMP generate reconstructed views with a higher quality than all other tested compression algorithms.

Infrared (IR) focal-plane array (FPA) detectors suffer from fixed-pattern noise (FPN), also known as spatial nonuniformity, which degrades image quality. In fact, FPN remains a serious problem despite recent advances in IRFPA technology. This work proposes a scene-based correction algorithm to continuously compensate for bias and gain nonuniformity in focal-plane array sensors. The proposed technique is a recursive algorithm based on recursive least square (RLS) techniques that jointly compensates for both bias and gain for each image pixel. The method converges rapidly and presents robustness to noise. Experiments with synthetic and real IRFPA videos has shown that it is competitive with the state-of-the-art in FPN reduction, presenting recovered images with higher fidelity when compared to them.

View synthesis from arbitrary viewpoints is an important component of virtual reality. This work addresses the problem of synthesizing a novel view from an arbitrary position given two images only (stereo pair). The proposed algorithm creates new viewpoints allowing six degrees of freedom to the virtual camera, i.e., translations along the three coordinate axis, pan, tilt and rotation around its own optical axis. The method generates new views for a totally non-calibrated stereo system. The novelty of the method relies in the placement of the virtual camera and its effect on the view synthesis. The synthesized views showed good results when compared with views produced by well-known viewpoint synthesis methods.

To characterize edges, this work proposes a zero-crossing detector that attempts to provide a more precise localization of edges together with robustness to brightness and contrast variation in images. Since the algorithm aims to be insensitive to image histogram manipulations, it is based on zero-crossings of the second derivative, and not on first derivative measurements. In order to provide good localization while preserving the true shape of the edges, the detector is based on precise mathematical expressions that represent image derivatives as a vector field, instead of using the Laplacian as a scalar approximation for the image second derivatives. This way, the proposed detector can be viewed as a Marr-Hildreth operator with a more exact mathematical formulation. Finally, to avoid false responses and provide thin edges, some additional checking is performed over the edge candidate points. The method showed good results when compared with well-known edge detectors.

For the compression of stereo images, several techniques employing the concept of disparity compensation have been developed. In this paper, we present a new method for stereo-image compression which is based on a different paradigm. It relies on approximate matching of multiscale recurrent patterns, which is an extension of ordinary pattern matching, where an image block is represented using dilated, contracted, displaced, and deformed versions of blocks already processed. To achieve this, the input image is segmented into variable-sized blocks. Each segment is represented by an element in a dictionary. The segmentation is performed according to a rate /spl times/ distortion criterion, and the dictionary is updated with the concatenation, contraction, expansion, displacement, and deformation of previously coded elements. Unlike most other methods, in this approach the disparity map is not transmitted. That is, the generation of the error images, as well as their coding and transmission, is not necessary. In brief, the proposed method presents high-quality results at the decoder end, compared to other coding methods for stereo images that transmit the disparity map.

A stereo image coding method using multiscale recurrent patterns is proposed. The method relies on the matching of recurrent patterns. The input image is segmented in variable-sized blocks. Each segment is coded by making use of contractions, expansions and displacements of elements of a dictionary. The segmentation is ruled by a rate distortion criterion and the dictionary is updated with the concatenation of previously coded elements. The novelty of this work is the absence of the disparity map in the stereo image coding process. That is, the burden of its calculation, the pre-and post-processing phases, the generation of the error images, as well as their coding and transmission are not necessary. In brief, the proposed method cuts off the load of the whole disparity estimation process yet presenting high-quality results at the decoder end.

The doubling of the data rate in stereo image systems causes problems for archiving and transmission, and compression is obviously desirable. In general, stereo codecs code one of the views as the reference using a conventional method like MPEG-2 assuming two basic sources, and the disparity map is calculated to aid the coding process. It can be coded using the DCT, another lossy method, or a lossless compression. The other view is either coded directly, or is restored from a combination of the reference view, the disparity data and a residual error image. In transform coding, inappropriate or coarse quantisation may cause coding artifacts. This work analyses and models the distribution of the 2D DCT coefficients of disparity maps and of disparity compensated residuals in the presence of occlusions, highlighting some points that should be considered when going from the monocular to the multiple-camera codec design process.

Multimedia and virtual reality systems give the user the illusion of three-dimensional contact. Unsurprisingly, this requires the transmission of more data than the monocular case. To bring participants together in a virtual space, or to allow a user to "see around" a 3-D object it is necessary to synthesise intermediate views from a stereoscopic image pair or sequence. In a stereo/video coding scheme, a block-based disparity map is usually coded and transmitted. This format is economic in computation and transmission, but likely to cause artifacts on any reconstructed intermediate view. To minimise the distortion of the reconstructed intensities, a nonlinear interpolator, which uses the single disparity available to each block of the image, generates a full-sized map that partially preserves original disparity edge information, producing a sharper intermediate view. The identification of occluded and non-occluded areas is also used to aid the intermediate view reconstruction.

The objective of stereo matching is, given a point in one view of a scene, to find the homologous point in another view from an adjacent camera. We use the discrete cosine transform (DCT) with evolutionary methods as an approach to solve the correspondence problem. The matching process is implemented by generating for each image block sets of DCT coefficients, which when reverse transformed become the estimated disparity map for that block. Assumed statistical properties of the AC coefficients are taken into account, while the DC component is estimated from a preliminary block matching stage. Matching is performed in the image domain using an intensity similarity measure. The job of optimising the DCT coefficients, whose inverse transform gives the disparity, is carried out by a biologically inspired optimisation technique: the genetic algorithm.