We propose a learning-based method for snapshot hyper-spectral (HS) imaging of deep 3D scenes. The method combines computational HS imaging and extended depth of field (EDoF) imaging capabilities in a single framework, resulting in novel EDoF-HS camera designs. The camera system incorporates a diffractive optical element at the aperture position, a CFA in front of the sensor and a residual dense network at the post-processing stage. These optical and neural components are jointly optimized through end-to-end learning procedure. We demonstrate high quality HS image reconstructions for scenes as deep as 4 diopters.

We present a learning model for reconstructing near-eye dense light field (LF) from a sparse set of multi-perspective views. The model integrates a fully-convolutional neural network and a model of the retinal image formation process optically connecting the pupil, retina and neural domains. Considering the problem of$9\times 9$near-eye LF reconstruction from the available five images, four at the corner viewpoints and one in the middle, we investigate the implications of using different loss functions in the learning process in terms of reconstruction qualities at different domains. Despite the utilized simplified retinal image formation model, the simulations reveal instructive results. In particular, combining the LF loss and the retinal focal stack loss is shown to improve the reconstruction quality of actual LF at the pupil plane, facilitating learning better features. On the other hand, concerning the retinal image quality, the model trained based on the same combination of losses is also demonstrated to produce better retinal images especially for non-Lambertian scenes, i.e., when there is monocular parallax, compared to model trained based on only retinal focal stack loss.

The efforts in compressive sensing (CS) literature can be divided into two groups: finding a measurement matrix that preserves the compressed information at its maximum level, and finding a robust reconstruction algorithm. In the traditional CS setup, the measurement matrices are selected as random matrices, and optimization-based iterative solutions are used to recover the signals. Using random matrices when handling large or multi-dimensional signals is cumbersome especially when it comes to iterative optimizations. Recent deep learning-based solutions increase reconstruction accuracy while speeding up recovery, but jointly learning the whole measurement matrix remains challenging. For this reason, state-of-the-art deep learning CS solutions such as convolutional compressive sensing network (CSNET) use block-wise CS schemes to facilitate learning. In this work, we introduce a separable multi-linear learning of the CS matrix by representing the measurement signal as the summation of the arbitrary number of tensors. As compared to block-wise CS, tensorial learning eases blocking artifacts and improves performance, especially at low measurement rates (MRs), such as [Formula: see text]. The software implementation of the proposed network is publicly shared at https://github.com/mehmetyamac/GTSNET.

We present a computational accommodation-invariant near-eye display, which relies on imaging with coherent light and utilizes static optics together with convolutional neural network-based preprocessing. The network and the display optics are co-optimized to obtain a depth-invariant display point spread function, and thus relieve the conflict between accommodation and ocular vergence cues that typically exists in conventional near-eye displays. We demonstrate through simulations that the computational near-eye display designed based on the proposed approach can deliver sharp images within a depth range of 3 diopters for an effective aperture (eyepiece) size of 10 mm. Thus, it provides a competitive alternative to the existing accommodation-invariant displays.

Depth of field is an important factor of imaging systems that highly affects the quality of the acquired spatial information. Extended depth of field (EDoF) imaging is a challenging ill-posed problem and has been extensively addressed in the literature. We propose a computational imaging approach for EDoF, where we employ wavefront coding via a diffractive optical element (DOE) and we achieve deblurring through a convolutional neural network. Thanks to the end-to-end differentiable modeling of optical image formation and computational post-processing, we jointly optimize the optical design, i.e., DOE, and the deblurring through standard gradient descent methods. Based on the properties of the underlying refractive lens and the desired EDoF range, we provide an analytical expression for the search space of the DOE, which is instrumental in the convergence of the end-to-end network. We achieve superior EDoF imaging performance compared to the state of the art, where we demonstrate results with minimal artifacts in various scenarios, including deep 3D scenes and broadband imaging.

We present an accommodation-invariant computational neareye display based on the extended depth of field imaging. The eyepiece of the display consists of a diffractive optical element (DOE) that is used in tandem with a conventional refractive lens. The DOE is co-designed with the pre-processing convolutional neural network, which is analogous to the post-processing deblurring networks in image capture. We demonstrate through simulations that such system achieves accommodation-invariant imaging within 2 Diopters depth range without significantly sacrificing spatial resolution.

We address the problem of 3D scene rendering on near-eye integral imaging displays and evaluation of different rendering methods in terms of human perception. We compare three rendering techniques in terms of perceived spatial resolution at different focused depths, simulating the display in virtual environment and representing the eye through a thin-lens camera model.

We present a learning-based optimization framework for depth of field extension, combining rigorous modeling of coded aperture imaging system and convolutional neural network based deblurring. The coded mask discretization is defined for desired depth range using wave optics based imaging model. Such approach significantly decreases the number of parameters to be optimized and increases the convergence speed of the network. We verify the proposed algorithm in different scenarios achieving superior or comparable performance with respect to existing methods.

Holographic stereograms (HSs) constitute one of the most widely used types of computer-generated holograms. The scene information required to calculate the HSs can be acquired by conventional digital cameras. It is, however, usually required that the scene should be captured from dense set of view points. Therefore, relieving this requirement is critical in the sense of easing the capture process. In this paper, in the capture stage of holographic stereograms, we employ our previously presented light field reconstruction algorithm [1], where we utilize sparse representation of light fields in the shearlet domain and reconstruct dense light fields from their highly under-sampled versions. The simulation results demonstrate that we can relieve the dense view sampling requirement of HSs, e.g. by as high as 8 × 8 sub-sampling factor, and still keep the perceived image quality of holographic reconstructions at satisfactory levels. This enables, for example, replacing the scanning camera setups with the more convenient multi-camera arrangements.

We investigate possible improvements that can be achieved in depth estimation by merging coded apertures and stereo cameras. We analyze several stereo camera setups which are equipped with different sets of coded apertures to explore such possibilities. The demonstrated results of this analysis are encouraging in the sense that coded apertures can provide valuable complementary information to stereo vision based depth estimation in some cases. In addition to that, we take advantage of stereo camera arrangement to have a single shot multiple coded aperture system. We show that with this system, it is possible to extract depth information robustly, by utilizing the inherent relation between the disparity and defocus cues, even for scene regions which are problematic for stereo matching.