Light field rendering (LFR) has been widely used for generating multiview images in Image-Based Rendering. However, to ensure the quality of novel views, this conventional rendering technology, requires a mass of manually captured light field images as input, which contains complete light field signal information and is highly time-consuming. Depth information of 3D scene and light field spectrum analysis are fundamental to light field reconstruction. The depth information was acquired with spectrum statistical analysis. This study aims to provide a broad applicability and robust depth layered scene mapping method by extending the traditional plenoptic sampling theory to reasonably determine the number of necessary captured images. A mathematical function was derived on the basis of overlapped spectrum, to decide the optimal mapping layer. Then, an optimal reconstruction filter is designed to reconstruct novel views. The proposed method exhibited a larger PSNR value in a variety of scenes. The depth layered scene mapping method showed potential for light field rendering without the need for complete multiview images requiring continuous sampling.

The aerial light field (ALF) can render higher quality images of large-scale 3D scenes. In this paper, we apply the ALF technology to study the image captured and novel view rendering of unmanned aerial vehicle (UAV), which exists some problems, such as large scene and depth of field. First, we design an ALF sampling model using spectral analysis of Fourier theory. Based on the ALF sampling model, the exact expression of ALF spectrum is derived. By the spectral support of ALF, we analyze the influence of pitch angle on light field sampling and its bandwidth. Particularly, the bandwidth of ALF can be applied to determine the minimum sampling rate for UAV. Additionally, we design a reconstruction filter that is related to pitch angle to render novel views of UAV. Finally, our experiments show that our sampling and rendering methods can improve the rendering quality of UAV novel view rendering.

In the previous results, light field sampling is based on ideal assumptions (e.g., Lambertian and Non-occluded scene), and thus we would like to more precisely analyze the sparsity sampling of light field signal. We present a sparsity analysis of light field (SALF) method for optimizing light field sampling rate. The SALF method applies the Fourier projection-slice theorem to simplify the initialization of light field sampling. Furthermore, we use a voting scheme to select light field spectra in which the frequency coefficients are nonzero. These spectra include many scene information and their captured positions are approximately equal to camera positions in the frequency domain. If the camera is only placed in these selected camera positions, the sampling rate can be optimized and the rendering quality can be guaranteed. Finally, we compare SALF method with other light field sampling methods to verify the claimed performance. The reconstruction results show that the SALF method improves rendering quality of novel views and outperforms those of other comparison methods.

Visual occlusion of scene remains one of the most difficult challenges in plenoptic spectral anaylsis, especially in the case of mutual occlusion. This paper introduces a new piecewise segmentation occlusion model for image-based plenoptic spectral analysis by dividing occlusion and nonocclusion span. Compared to other works on plenoptic spectral analysis, we consider the fundamental problem of occlusion, where the light ray emitted from the object surface are analyzed from camera plane and image plane separately. We first treat the light field using Fourier transform equation, including occlusion and nonocclusion integral span. We then estimate the plenoptic bandwidth and the minimum sampling rate of the POF. We demonstrate results for various scenes, showing the variation in the occlusion spectrum relative to the nonocclusion spectrum and the consistency between the theoretical analysis and the simulation results. Furthermore, our reconstruction simulation result enables us to show that our framework performs well within classical method.

Occlusions are common phenomena in light field rendering (LFR) technology applications. The 3-D spatial structures of some features may be missing or incorrect when capturing some samples due to occlusion discontinuities. Most prior works on LFR, however, have neglected occlusions from other objects in 3-D scenes that do not participate in the capturing and rendering of the light field. To improve rendering quality, this report proposes an occlusion probability learning framework (OPLF) based on a deep Boltzmann machine (DBM) to compensate for the occluded information. In the OPLF, an occlusion probability density model is applied to calculate the visibility scores, which are modeled as hidden variables. Additionally, the probability of occlusion is related to the visibility, the camera configuration (i.e., position and direction), and the relationship between the occlusion object and occluded object. Furthermore, a deep probability model based on the OPLF is used for learning the occlusion relationship between the camera and object in multiple layers. The proposed OPLF can optimize the LFR quality. Finally, to verify the claimed performance, we also compare the OPLF with the most advanced occlusion theory and light field reconstruction algorithms. The experimental results show that the proposed OPLF outperforms other known occlusion quantization schemes.

A number of theories have been proposed for reducing sampling rate of light field. But these theories still need a great many of samples (images) to obtain sufficient geometric information. In this paper, we utilize the sparse representation of light field in Discrete Cosine Transform domain to present a Discrete Cosine Sparse Basis (DCSB). Thus, we can find out the zeros of DCSB to reduce sampling requirement of light field for alias-free rendering. Finally, experimental results demonstrate the effectiveness of our approach without lose information.

Occlusion lack compensation (OLC) is a multiplexing gain optimization data acquisition and novel views rendering strategy for light field rendering (LFR). While the achieved OLC is much higher than previously thought possible, the improvement comes at the cost of requiring more scene information. This can capture more detailed scene information, including geometric information, texture information and depth information, by learning and training methods. In this paper, we develop an occlusion compensation (OCC) model based on restricted boltzmann machine (RBM) to compensate for lack scene information caused by occlusion. We show that occlusion will cause the lack of captured scene information, which will lead to the decline of view rendering quality. The OCC model can estimate and compensate the lack information of occlusion edge by learning. We present experimental results to demonstrate the performance of OCC model with analog training, verify our theoretical analysis, and extend our conclusions on optimal rendering quality of light field.

Phase unwrapping is a fundamental task in phase‐based profilometry. Existing spatial and temporal approaches are facing challenges such as error propagation and low efficiency. In this study, the authors propose a learning‐based method that uses a support vector machine (SVM) to perform phase unwrapping, where the problem is solved as a classification task. To be specific, seven elements, extracted from the captured patterns and the wrapped phase, form the input feature vector and the fringe order is the output class. Besides, a radial basis function kernel SVM is adopted as the model. The proposed method is conducted independently for every pixel, and does not suffer from error propagation in the spatial unwrapping. Moreover, it needs fewer patterns than temporal unwrapping since only one phase map is required. Simulation and experimental results demonstrate that the proposed scheme produces precise depth maps, which are comparable with the complex quality‐guided methods but at a much faster speed.

Occlusions will reduce the performance of systems in many computer vision applications with discontinuous surfaces of 3D scenes. We explore a signal-processing framework of occlusions based on the light ray visibility to improve the rendering quality of views. An occlusion field (OCF) theory is derived by calculating the relationship between the occluded light rays and the nonoccluded light rays to quantify the occlusion degree (OCD). The OCF framework can describe the various in-scene information captured by the changes in the camera configuration (i.e., position and direction) through a quantitative description of the occlusion information. From a spectral analysis of the OCF, we mathematically derive analytical functions to determine the changing relationship between the scene and the camera configuration. A reconstruction filter can be designed to achieve interference cancellation and compensate for the missing information caused by the occlusions. Our measurements of different occlusions using this OCF framework included both synthetic and actual scenes. The experimental results show that the proposed OCF framework can improves the rendering quality of views and outperforms other known occlusion quantization schemes in a complex scene.

Occlusion as a common phenomenon in object surface can seriously affect information collection of light field. To visualize light field data-set, occlusions are usually idealized and neglected for most prior light field rendering (LFR) algorithms. However, the 3D spatial structure of some features may be missing to capture some incorrect samples caused by occlusion discontinuities. To solve this problem, we propose an occlusion probability (OCP) model to improve the capturing information and the rendering quality of views with occlusion for the LFR. In this OCP model, a probability density model is applied to obtain the scores of visibility are modeled as hidden variables. The occlusion probability is calculated by the visibility, position and orientation of camera. We compare different capturing/reconstruction techniques to visualize/manipulate our OCP model.

Rendering quality optimization theory is one of the most basic and fascinating components of image-based rendering (IBR). The rendering quality of virtual views is related to the information in the source images. Capturing the image information depends on the geometric configuration of the camera (GCC) and, in particular, the positions and shooting directions of the cameras. Therefore, the rendering quality of virtual views can be improved by optimizing the GCC. This paper investigates the relationship between the GCC and the geometric configurations of the virtual view (GCVV). The influence of the GCC on the rendering quality is also analyzed. Based on the relationships among the GCC, GCVV, and rendering quality, a mathematical model of the noncoverage field (NCF) is proposed to quantify the rendering quality. The performance of the NCF is also analyzed using a set of GCVVs and GCCs. Furthermore, an optimization algorithm based on the NCF that simultaneously optimizes the position and direction of the GCC is presented. The proposed technique can be applied to obtain the optimal rendering quality of IBR for the linear case of camera positions and virtual views. Finally, experimental results are presented and compared with the theoretical results.

Scene surface exists some occlusions by mutual occlusions or self-occlusions, and then this phenomenon will serious influences 3D vision technique. We present a signal-processing framework to study occlusions of scene. Our method combines discontinuities and establish a mathematical model of occlusion phenomenon. The influences of occlusion on a scene are derived using Fourier theory. In this manner, spectral support of scene occlusion is analyzed in frequency field. Predictions on the frequency content can then be used to control the sampling and rendering in a scene. This extends previous work that the sampling and rendering to be analyzed and estimated for 3D technique. Extensive experimental evaluation demonstrates that our occlusion analysis method significantly outperform competing algorithms.

In this paper, a new method is proposed to generate image-assisted geometry simplification for estimating minimum sampling rate of image-based rendering (IBR). If some geometry information (e.g., the shape of object surface and proxy geometry) on a scene is known, we can decompose the scene geometry into a collection of simpler structures on a block-by-block basis. Our framework predicts the characteristics of simpler structure such as irregular object. Predictions on the frequency content can then be used to control sampling rates for IBR. The new method allows the sampling of the IBR to be analyzed and estimated for the non-uniform sampling. Furthermore, the minimum sampling rate of the IBR necessary for alias-free rendering will be reduced as the number of simpler structures increases.