Light field (LF) rendering is widely used in free viewpoint video systems (FVV). Different methods have been proposed to employ depth maps to improve the rendering quality. However, estimation of depth is often error-prone. In this paper, a new method based on the concept of effective sampling density (ESD) is proposed for evaluating the depth-based LF rendering algorithms at different levels of errors in the depth estimation. In addition, for a given rendering quality, we provide an estimation of number of rays required in the interpolation algorithm to compensate for the adverse effect caused by errors in depth maps. The proposed method is particularly useful in designing a rendering algorithm with inaccurate knowledge of depth to achieve the required rendering quality. Both the theoretical study and numerical simulations have verified the efficacy of the proposed method.

In a light field-based free viewpoint video (LF-based FVV) system, effective sampling density (ESD) is defined as the number of rays per unit area of the scene that has been acquired and is selected in the rendering process for reconstructing an unknown ray. This paper extends the concept of ESD and shows that ESD is a tractable metric that quantifies the joint impact of the imperfections of LF acquisition and rendering. By deriving and analyzing ESD for the commonly used LF acquisition and rendering methods, it is shown that ESD is an effective indicator determined by system parameters and can be used to directly estimate output video distortion without access to the ground truth. This claim is verified by extensive numerical simulations and comparison to PSNR. Furthermore, an empirical relationship between the output distortion (in PSNR) and the calculated ESD is established to allow direct assessment of the overall video distortion without an actual implementation of the system. A small scale subjective user study is also conducted which indicates a correlation of 0.91 between ESD and perceived quality.

Acquisition of a free viewpoint video (FVV) system is theoretically considered as a problem of plenoptic signal sampling. It is typically performed by using a regular camera grid. While a regular acquisition itself results in non-uniform sampling density, this non-uniformity does not match the scene complexity and frequency variations. This paper proposes an irregular acquisition method for optimum non-uniform plenoptic sampling corresponding to the variations of the scene complexity. Specifically, scene complexity is measured through analyzing DCT coefficients of reference images of the scene, describing the frequency behavior of the plenoptic signal over the scene space. An optimization model is formulated to calculate the optimum configurations of the acquisition cameras including positions and orientations. The theoretical analysis and numerical simulations demonstrate that the rendered video quality can be significantly improved (around 40% in mean PSNR) by employing the proposed irregular acquisition compared with the regular camera grid.

In a free viewpoint video system, the scene is captured by a number of cameras and it would be desirable to optimize the configuration of cameras, such as their location or orientation, to improve the rendering quality. This paper introduces a mathematical representation of the multi-camera geometry, called the correspondence field (CF), which can be used to quantify the suitability of a camera configuration for a given arrangement of objects in the scene. The correspondence field describes the spatial topology of the intersecting rays of cameras, arranged as a number of layers or surfaces in the field of view of cameras. The paper derives the topology of CF for certain camera arrangements and analyzes the impact of changes in camera location or orientation on this topology. It demonstrates that CF can be used to find the optimum camera configuration for a given objective. It also presents simulation results of this method using our light field simulator.

Calculation of the number of cameras required to capture the scene is an essential problem in a practical light field based free viewpoint video (FVV) system. Existing methods calculate the Nyquist rate by assuming a band-limited signal and perfect reconstruction of an arbitrary view using linear interpolation, which often results in an impractically high number of cameras. This paper proposes a new method based on the concept of effective sampling density (ESD). It is demonstrated that there is a trade-off between the depth information accuracy, the required number of cameras, and the desired rendering quality, which could be exploited to minimize the number of cameras for a given objective. Theoretical and numerical results show that the resulting number of cameras is significantly lower than what was reported in other studies with only a few percent reduction in the output PSNR. Moreover, it is shown that the methods proposed in those studies are special cases of the one presented in this paper.

Light field rendering (LFR) is an active research area in computer vision and computer graphics. LFR plays a crucial role in free viewpoint video systems (FVV). Although several rendering algorithms have been suggested for LFR but the lack of appropriate datasets with known ground truth has prevented a comparison and evaluation study of LFR algorithms. In most of the LFR papers the method is applied to several test cases for validation and as a result, just a subjective visualized output is given. To overcome this problem, this paper presents a quantitative approach for comparison and evaluation of LFR algorithms. The core of the proposed methodology is a simulation model and a 3D engine. The platform produces the reference images and ground truth data for a given 3D model. Subsequently, data are injected to a comparison engine to compare synthesized images from light field engine with original images from simulation, generating objective results for evaluation. The methodology is highly flexible and efficient to automatically generate LFR datasets and objectively compare and analyze any subset of LFR methods for any given experiment design scheme. Five key rendering algorithms are evaluated with proposed methodology to validate it. Overall, it is shown that the proposed quantitative methodology could be used for LFR objective evaluation and comparison.

Light field rendering (LFR) is an active research area in computer vision and computer graphics. LFR plays a crucial role in free viewpoint video systems (FVV). Several rendering algorithms have been suggested for LFR. However, comparative evaluation of these methods is often limited to subjective assessment of the output. To overcome this problem, this paper presents a geometric measurement, Effective Sampling Density of the scene, referred to as effective sampling for brevity, for objective comparison and evaluation of LFR algorithms. We have derived the effective sampling for the well-known LFR methods. Both theoretical study and numerical simulation have shown that the proposed effective sampling is an effective indicator of the performance for LFR methods.