Recent dense multi-view 3D systems may provide consistently augmented reality, where we enjoy a simple solution of occlusion caused by visual obstructions in the real world, as if they were transparent for many users at the same time. In this paper, we propose a novel real-time method to reconstruct dense 4D light fields from RGB-D data for such virtual transparency of visual obstructions. First, we discuss appearances of our proposed multi-view system, while examining how densely 4D light fields should be reproduced. Then, in order to develop an inexpensive system, we propose real-time reconstruction of the entire 4D light field from point cloud data acquired by a single RGB-D camera instead of massive camera arrays. Actually, by applying appropriate geometrical transformation to the acquired RGBD data, we obtain a simple 3D array while effectively reducing their redundancy. Moreover, it allows us to achieve structured analysis of the scene for reconstructing a 4D light field composed of equally-spaced multi-view images, where its computational cost is significantly suppressed.

We previously implemented an inexpensive imaging system that combines a single real camera with a mirror array located along a paraboloid. It allows us to robustly acquire dynamic light fields composed of multi-view videos by providing a virtual camera array, where its viewpoints exist in the mirrors. Actually, as moving the real camera to the focus of the paraboloid, virtual viewpoints in the mirrors get equally-spaced to achieve multi-view imaging with structured disparity. In this paper, we discuss an efficient method for adjusting the pose of a single camera to acquire high quality dynamic light fields as multi-view videos. Specifically, we introduce some indicator values determined by detected corners of the mirror array on acquired images while adjusting the camera. By using these values for camera adjustment, we easily know how to move its position and virtually correct its angle through homography transform. Experimental results of simulations demonstrate that our proposed method sufficiently achieves structured light field video acquisition with equally-spaced virtual viewpoints, where we do not need camera rotation requiring complex devices and only the camera position is controlled by a simple 3D system like XYZ stages.

In this study, we propose an iterative approach to perform pseudo-scene reconstruction based on multi-focus images for estimating the 4D light fields of the scene robustly. An analogous scene-independent linear filtering approach has been proposed to efficiently achieve free viewpoint image reconstruction and scene refocusing with virtual bokeh by solely extracting 3D frequency components. However, it only succeeded in synthesizing limited regions of 4D light fields from multi-focus images, which can lead to awful degradation of reconstructed free viewpoint images outside the range due to the lack of the corresponding frequency components. In this paper, we directly utilize a linear relationship of 3D convolution combining multi-focus images and 3D scene by 3D blurring filters to estimate the 3D distribution itself of the scene instead of deriving the scene-independent linear filters from the relationship. In other words, we perform the scene reconstruction as a simplified convex optimization problem by modeling multi-focus images in multi-layered blurring composed of the scene and a bokeh filter. Algebraic reconstruction techniques of computed tomography, which uses the classical steepest descent method combined with projection onto convex sets (POCS) to ensure the convexity of the problem constraints, are introduced into such 3D scene reconstruction. The new method can be highly anticipated to assist us in extensively estimating robust 4D light fields. Experimental results illustrate that this approach enables good quality 3D image reconstruction with a wider range of 4D light fields. We also discuss the reduction of high computational costs and enormous memory requirements imposed by iterative algebraic reconstruction techniques. The quality of the obtained 4D light fields is comprehensively evaluated using both synthetic and real multi-focus images of practicable image resolution and depth range, additionally to an analysis of the faster processing times when comparing our iterative approach to the performance of the prior linear filters.

A light field (LF) image is composed of multiple sub-aperture images (SAIs) from slightly offset viewpoints. To denoise a noise-corrupted LF image, leveraging recent development in deep algorithm unfolding, we pursue a hybrid graph-model-based / data-driven approach. Specifically, we first connect each pixel in a target patch of an SAI to neighboring pixels within the patch, and to pixels in co-located "similar" patches in adjacent SAIs. Given graph connectivity, we formulate a maximum a posteriori (MAP) problem using graph total variation (GTV) as signal prior. We then unroll the iterations of a corresponding optimization algorithm into a sequence of neural layers. In each unrolled layer, we learn relevant features per pixel from data using a convolutional neural net (CNN) in a supervised manner, so that edge weights can be computed as functions of feature distances. Each neural layer can be interpreted as a graph low-pass filter for a 4D LF image patch. Experiments show that our proposal outperformed two model-based and two deep-learning-based implementations in numerical and visual comparisons.

We propose a novel method of light field compression using multi-focus images and reference views. Light fields enable us to observe scenes from various viewpoints. However, it generally consists of 4D enormous data, that are not suitable for storing or transmitting without effective compression at relatively low bit-rates. On the other hand, 4D light fields are essentially redundant because it includes just 3D scene information. While robust 3D scene estimation such as depth recovery from light fields is not so easy, a method of reconstructing light fields directly from 3D information composed of multi-focus images without any scene estimation is successfully derived. Based on the method, we previously proposed light field compression via multi-focus images as effective representation of 3D scenes. Actually, its high performance can be seen only at very low bit-rates, because there exists some degradation of low frequency components and occluded regions on light fields predicted from multi-focus images. In this paper, we study higher quality light field compression by using reference views to improve quality of the prediction from multi-focus images. Our contribution is twofold: first, our improved method can keep good performance of 4D light field compression at a wider range of low bit-rates than the previous one working effectively only for very low bit-rates; second, we clarify how the proposed method can improve its performance continuously by introducing recent video codec such as HEVC and VVC into our compression framework, that does not depend on 3D-SPIHT previously adopted for the corresponding component. We show experimental results by using synthetic and real images, where quality of reconstructed light fields is evaluated by PSNR and SSIM for analyzing characteristics of our novel method well. We notice that it is much superior to light field compression using HEVC directly at low bit-rates regardless of its light field scan order.

We propose a novel method of light field denoising for 3D consistent visualization. Light field data are often significantly degraded by awful noise due to lack of exposure under insufficient lighting, especially, if acquired by a lens array. Then, recent studies notice that a 4D light field corresponds to structured multi-view images that are correlated to each other well according to 3D scenes. For example, a CNN-based denoising method utilizes the correlation by analyzing various subspaces of 4D light fields. In this paper, 4D light field denoising is achieved by applying simple restoration based on Total Variation minimization to small images composed of corresponding pixels extracted from multi-view images. We show experimental results using synthetic and real images to clarify how our proposed method effectively works for 3D consistent visualization in comparison with the other methods. In addition, we discuss possible future improvement of the proposed method when integrating our novel approach with the conventional 2D image denoising for more effective light field denoising.

Lens arrays enable us to easily achieve inexpensive light field acquisition in comparison with massive camera arrays. However, their micro-lenses often cause insufficient exposure to obtain high quality multi-view videos like camera arrays. In this paper, we propose a novel virtual camera array acquiring dynamic light fields as multi-view videos robustly and inexpensively, where mirror arrays consisting of tangent planes to a paraboloid are effectively combined with a single video camera. First, we demonstrate multi-view imaging of not an object but ordinary scenes with sufficient exposure by using a simple horizontal mirror array implemented as a prototype. Next, we theoretically reveal that there exists smooth extension of our mirror array to the vertical direction, that can be implemented easily by 3D printing, CNC machines, and so on. Finally, our design of light field acquisition is improved for effectively utilizing the whole of its mirror array by introducing an one-way mirror to prevent the camera itself from appearing on multi-view images.

In this paper, we propose a novel method restoring multi-focus images based on convex optimization with new constraint for fixed pattern noise. Even weak fixed pattern noise on multi-focus images degrades all-in-focus images reconstructed by linear combination of them, especially, when using telecentric optical systems such as microscopes. Our novel method introduces constraint for additive fixed pattern noise into total variation minimization and then it is improved for multiplicative fixed pattern noise. The proposed method suppresses fixed pattern noise on multi-focus images very robustly to avoid such degradation on reconstructed images. Experimental results show that our method achieves high performance compared to simple total variation minimization.

We previously proposed a simple method of 3-D image reconstruction using scene-independent linear filters for multi-focus images. However, it costs much to prepare the linear filters for high-resolution images. In this paper, by eliminating symmetric redundancy of the linear filtering, we significantly reduce required memory and computational cost by less than one-tenth. Then, we achieve real-time and interactive image reconstruction with 3-D effects from high-resolution multi-focus images on a GPGPU. Experimental results using microscopic images reveal that our novel method enables us to observe 3-D objects very effectively by free viewpoint image reconstruction and scene refocusing with desired bokeh at about 60 fps and 15 fps, respectively.

If scene flow expressed in three-dimensional (3D) vector fields is robustly estimated from multi-view or multi-focus images, we can develop advanced 3D motion tracking and motion compensation for 3D video compression. In this study, based on a synthesis of multi-focus images from multi-view images, we propose a novel method for analyzing 3D scene flow accurately at low computational cost as an extension of 2D optical flow estimation. Our method directly estimates 3D scene flow in multi-focus images arranged in a direction orthogonal to them by an extended pyramidal Lucas-Kanade method for 3D. Experimental results show that scene flow estimation is achieved efficiently when using our proposed method. In addition, our method can be easily implemented on a graphics processing unit for real-time applications.

Scene refocusing and free viewpoint image reconstruction from a dense light field acquired by a massive camera array or a lens array are actively studied. However, it is difficult to directly integrate acquired information into a light field without defects. For example, actually, a massive camera array hardly works without failure of several cameras, and part of multi-view images are missing. On the other hand, because a lens array consists of very small lenses gathering only limited rays, acquired images themselves are often noisy. This paper proposes simple linear filters achieving view/image restoration to obtain a dense light field robustly in suppressing such defects without block matching used by the conventional approaches of view synthesis and denoising. Experimental results show that the proposed filters can be flexibly designed for various defects based on the relation between a 4D light field and the corresponding 3D multi-focus images.

Denoising is important in image processing because degradation by noise affects not only the quality of captured images but also the performance of visual applications that use them. For example, under low light levels, it is difficult to accurately estimate scene depths using noisy stereo images. Conventional methods for denoising find similar regions on an image or among multiple images by block matching(BM) to integrate them for suppressing noise effectively. However, such exhaustive BM incurs considerable costs for real-time applications, in particular, when multi-view images(MVI) are involved. We use view-dependent plane sweeping(PS) for image reconstruction to achieve effective MVI denoising with low computational cost. We use PS for converting MVI to multi-focus images(MFI) to suppress their noise. Then, we find regions in focus on the MFI solely by comparing them with the target view image. Finally, we simply merge the regions to obtain reconstructed images in which their noise is effectively suppressed.

Scene refocusing beyond extended depth of field for users to observe objects effectively is aimed by researchers in computational photography, microscopic imaging, and so on. Ordinary all-in-focus image reconstruction from a sequence of multi-focus images achieves extended depth of field, where reconstructed images would be captured through a pinhole in the center on the lens. In this paper, we propose a novel method for reconstructing all-in-focus images through shifted pinholes on the lens based on 3D frequency analysis of multi-focus images. Such shifted pinhole images are obtained by a linear combination of multi-focus images with scene-independent 2D filters in the frequency domain. The proposed method enables us to efficiently synthesize dense 4D light field on the lens plane for image-based rendering, especially, robust scene refocusing with arbitrary bokeh. Our novel method using simple linear filters achieves not only reconstruction of all-in-focus images even for shifted pinholes more robustly than the conventional methods depending on scene/focus estimation, but also scene refocusing without suffering from limitation of resolution in comparison with recent approaches using special devices such as lens arrays in computational photography.

Light-Field enables us to observe scenes from free viewpoints. However, it generally consists of 4-D enormous data, that are not suitable for storing or transmitting without effective compression. 4-D Light-Field is very redundant because essentially it includes just 3-D scene information. Actually, although robust 3-D scene estimation such as depth recovery from Light-Field is not so easy, we successfully derived a method of reconstructing Light-Field directly from 3-D information composed of multi-focus images without any scene estimation. On the other hand, it is easy to synthesize multi-focus images from Light-Field. In this paper, based on the method, we propose novel Light-Field compression via synthesized multi-focus images as effective representation of 3-D scenes. Multi-focus images are easily compressed because they contain mostly low frequency components. We show experimental results by using synthetic and real images. Reconstruction quality of the method is robust even at very low bit-rate.

Light-Field enables us to observe scenes from free viewpoints. However, it generally consists of 4-D enormous data, that are not suitable for storing or transmitting without effective compression. 4-D Light-Field is very redundant because essentially it includes just 3-D scene information. Actually, although robust 3-D scene estimation such as depth recovery from Light-Field is not so easy, a method of reconstructing Light-Field directly from 3-D information composed of multi-focus images without any scene estimation is successfully derived. Previously, based on the method, Light-Field compression via synthesized multi-focus images as effective representation of 3-D scenes was proposed. In this paper, we study efficient compression of multi-focus images synthesized from dense Light-Field by using DWT instead of DCT-based compression in order to suppress degradation such as block noise. Quality of reconstructed Light-Field is evaluated by PSNR and SSIM for analyzing characteristics of residuals. Experimental results reveal that our method is much superior to Light-Field compression using disparity-compensation at low bit-rate.

We previously proposed a method of generating free viewpoint images directly from multi-focus imaging sequences without any depth estimation. It is very effective for the method to be implemented to hardware such as FPGA. However, the number of BlockRAMs on our FPGA limits the image size to 64 × 64 pixels. In this paper, we extend our FPGA-based free viewpoint image reconstruction systems by using an onboard DDR SDRAM and processing the divided blocks of 64 × 64 pixels repeatedly. The system realizes our proposed method even for larger image sizes without great drawbacks. Some experimental results by using synthetic images are shown.

This paper deals with a method of generating seriously deformed bokeh on reconstructed images from ordinary multiple differently focused images including just simple bokeh such as Gaussian blurs. We previously proposed scene re-focusing with various iris shapes by applying a three-dimensional filter to the multi-focus images. However, actually the proposed method implicitly assumed that the feature of the iris can be expressed mathematically and it has some symmetry like a horizontally open iris. In this paper, at first, the captured multi-focus images are robustly decomposed into components, each of which goes through its own corresponding pin-hole on the lens, by using dimension reduction and a two-dimensional filter. Then, based on the appropriate composition of the components, reconstruction of arbitrarily deformed bokeh introduced by any user-defined iris is achieved. By some experiments, we show that our novel method can generate even seriously deformed bokeh that does not have simple symmetry.

We have been investigating an image sensor in which the 2D motion vector of each pixel is detected at very high frame rates such as those greater than 1000 frames per second. In this sensor, the motion vector is detected by single pixel matching with a limited search range and high inter-frame correlation at a high frame rate. In this paper, we have proposed a new method of on-sensor motion detection for noise reduction and improvement of motion direction accuracy. In this method, we implemented two new functions on the sensor. We fabricated the prototype chip using a 0.35 um CMOS process. Furthermore, we explained the methods for estimating the motion trajectory of this imaging device by using the output from the sensor and for estimating the motion direction of a moving object along the depth direction; these two methods are the applications of this sensor. The simulation results show that the proposed on-sensor motion vector detection is more effective than our previously proposed method.

This paper deals with all-in-focus image reconstruction by merging multiple differently focused images. We previously proposed a method of generating an all-in-focus image from multi-focus imaging sequences based on a 3-D filtering. In this paper, we first combine the sequence into a 2-D image. Just by applying a 2-D filter to the image, we realize fast reconstruction of all-in-focus images robustly. In order to reduce the cost of image acquisition, the optimal number of multiple differently focused images is also discussed. In addition, we introduce a simple estimation method utilizing the 2-D filter for the parameter of 3-D blurs. We show experimental results of fast all-in-focus image reconstruction by using synthetic and real images.

This paper deals with a view interpolation problem using multiple images captured with a circular camera array. An inverse filtering method for reconstructing a virtual image at the center of the camera array is proposed. First, we generate a candidate image by adding all correspondence pixel values based on multiple layers assumed in a scene. Second, we model a linear relationship between the desired virtual image and the candidate image, and then derive the inverse filter to reconstruct the virtual image. Since we can derive the inverse filter independent of the scene structure, the proposed method requires no depth estimation. Simulation results using synthetic images show that increasing the number of cameras and layers improves the quality of the virtual image.

This paper describes a method of image generation based on transformation integrating certain sequences of multiple differently focused images. First, we assume that a scene is defocused by a geometrical blurring model. Then we combine spatial frequencies of the scene and the sequence with a 3-D convolution filter that expresses how the scene is defocused on the sequence. The filter can be represented with a linear combination of ray-sets through each point of the lens. Based on the relation, in the 3-D frequency domain we extract each ray-set from the filter as certain frequency components and merge them to reconstruct various filters that can generate images with different viewpoints and blurs.

This paper describes a method of scene re-focusing with various iris shapes by integrating a sequence of multiple differently focused images. First, we introduce a formula that combines the sequence and spatial information of a scene with a convolution of a 3-D blur. The blur expresses how the scene is defocused in the sequence. Based on the formula, in the 3-D frequency domain we can design filters merging certain frequency components of the sequence to generate images that would be acquired with various iris shapes. Some experiments of scene re-focusing using synthetic and real images indicate that we can not only arbitrarily suppress blurs of the sequence but also generate images with asymmetrical blurs like motion blurs.

This paper deals with a view interpolation problem using multiple images captured with a circular camera array. A novel deconvolution method for reconstructing a virtual image at the center of the camera array is presented and discussed in a framework of image restoration in the frequency domain. The reconstruction filter does not depend on the scene structure; therefore the presented method requires no depth estimation. The simulation result using synthetic images shows that increasing the number of the cameras improves the quality of the virtual image.

Construction of large-scale virtual environment is gaining more attention for its applications in virtual malls, virtual sightseeing, tele-presence, etc. We introduce a framework to construct a realistic large-scale virtual environment by fusing range data, texture images, and airborne altimetry data. First, realistic and high-precision 3-D models of buildings are created from range data and texture images, which are taken by long-range laser scanner and digital camera, respectively. Next, rough 3-D models of buildings in wide area are created using altimetry data acquired by airborne laser profiler. Finally, these models are integrated to build a large-scale realistic walk-through system. This paper describes the proposed system, issues related to the system, part of its implementations, as well as future work that still has to be done.

We propose a novel approach for producing special visual effects by fusing multiple differently focused images. This method differs from conventional image fusion techniques because it enables us to arbitrarily generate object-based visual effects such as blurring, enhancement, and shifting. Notably, the method does not need any segmentation. Using a linear imaging model, it directly generates the desired image from multiple differently focused images.

We propose a novel approach to image content manipulation. It enables us to arbitrarily manipulate an object in a scene by linear processings such as blurring, enhancement and shift etc. Notably, the method does not need any segmentation nor 3D modeling. Making use of multiple differently focused images and a linear imaging model, it directly generates the desired image from the original images. A special camera is developed which can acquire three differently focused image sequences, in order to extend the proposed method to image sequence processing.

In this paper, we propose a registration method between multiple differently focused images using the hierarchical block matching technique in which displacement, scale and rotation are taken into account. Local deformation due to lens distortion is further corrected by local matching. We also propose an efficient estimation method of blur parameters of defocused regions in these focused images. Simulation results showed that the proposed methods achieve high accuracy. In experiments using real images captured by hand-held camera, an all focused image with good quality was able to be automatically generated using the corrected images and the estimated parameters.

In this paper, we propose a novel method of all-focused image acquisition using multiple differently focused images. Based on the assumption that depth of the scene changes stepwise, we derive a formula for reconstruction between the desired all-focused image and multiple acquired images; we can reconstruct the all-focused image by iterative use of the formula. We also introduce coarse-to-fine estimation of point spread functions of the acquired images. We show we can reconstruct an all-focused image for a natural scene.