Although context-based monocular depth estimation has shown remarkable improvement, the adaptation to unseen contexts is still a major challenge. On the other hand, the use of physical depth cues, such as defocus associated with lens aberration, allows context-independent depth estimation. However, explicitly supervising physical depth cues would have a significant impact on cost and versatility, because of the need to use expensive equipment to obtain the ground truth. Therefore, we propose a novel self-supervised learning for a single-shot neural depth from defocus (DfD) utilizing structure from motion (SfM) images taken by the target lens. Since the scale of SfM depth is ambiguous, we used rank loss to train the network. To demonstrate the versatility of our method, we conducted validation experiments using not only DSLR cameras but also smartphones with small image sensors. We confirmed that our method is highly accurate by a large margin over state-of-the-art methods including the physically-calibrated neural single-shot DfD and context-based methods.

This paper presents a novel active marker for dynamic projection mapping (PM) that emits a temporal blinking pattern of infrared (IR) light representing its ID. We used a multi-material three dimensional (3D) printer to fabricate a projection object with optical fibers that can guide IR light from LEDs attached on the bottom of the object. The aperture of an optical fiber is typically very small; thus, it is unnoticeable to human observers under projection and can be placed on a strongly curved part of a projection surface. In addition, the working range of our system can be larger than previous marker-based methods as the blinking patterns can theoretically be recognized by a camera placed at a wide range of distances from markers. We propose an automatic marker placement algorithm to spread multiple active markers over the surface of a projection object such that its pose can be robustly estimated using captured images from arbitrary directions. We also propose an optimization framework for determining the routes of the optical fibers in such a way that collisions of the fibers can be avoided while minimizing the loss of light intensity in the fibers. Through experiments conducted using three fabricated objects containing strongly curved surfaces, we confirmed that the proposed method can achieve accurate dynamic PMs in a significantly wide working range.

Passive and convenient depth estimation from single-shot image is still an open problem. Existing depth from defocus methods require multiple input images or special hardware customization. Recent deep monocular depth estimation is also limited to an image with sufficient contextual information. In this work, we propose a novel method which realizes a single-shot deep depth measurement based on physical depth cue using only an off-the-shelf camera and lens. When a defocused image is taken by a camera, it contains various types of aberrations corresponding to distances from the image sensor and positions in the image plane. We call these minute and complexly compound aberrations as Aberration Map (A-Map) and we found that A-Map can be utilized as reliable physical depth cue. Additionally, our deep network named A-Map Analysis Network (AMA-Net) is also proposed, which can effectively learn and estimate depth via A-Map. To evaluate validity and robustness of our approach, we have conducted extensive experiments using both real outdoor scenes and simulated images. The qualitative result shows the accuracy and availability of the method in comparison with a state-of-the-art deep context-based method.

One of the solutions of depth imaging of moving scene is to project a static pattern on the object and use just a single image for reconstruction. However, if the motion of the object is too fast with respect to the exposure time of the image sensor, patterns on the captured image are blurred and reconstruction fails. In this paper, we impose multiple projection patterns into each single captured image to realize temporal super resolution of the depth image sequences. With our method, multiple patterns are projected onto the object with higher fps than possible with a camera. In this case, the observed pattern varies depending on the depth and motion of the object, so we can extract temporal information of the scene from each single image. The decoding process is realized using a learning-based approach where no geometric calibration is needed. Experiments confirm the effectiveness of our method where sequential shapes are reconstructed from a single image. Both quantitative evaluations and comparisons with recent techniques were also conducted.

The central projection model commonly used to model cameras as well as projectors, results in similar advantages and disadvantages in both types of system. Considering the case of active stereo systems using a projector and camera setup, a central projection model creates several problems, among them, narrow depth range and necessity of wide baseline are crucial. In the paper, we solve the problems by introducing a light field projector, which can project a depth-dependent pattern. The light field projector is realized by attaching a coded aperture with a high frequency mask in front of the lens of the video projector, which also projects a high frequency pattern. Because the light field projector cannot be approximated by a thin lens model and a precise calibration method is not established yet, an image-based approach is proposed to apply a stereo technique to the system. Although image-based techniques usually require a large database and often imply heavy computational costs, we propose a hierarchical approach and a feature-based search for solution. In the experiments, it is confirmed that our method can accurately recover the dense shape of curved and textured objects for a wide range of depths from a single captured image.

Summary form only given. In this tutorial, we will introduce emerging technologies on computational imaging and light field projection to AR/MR researchers.Light is the most important medium in AR/VR technologies to not only obtain information but also show and modify visual cue in the real scenes. Therefore in this area, latest techniques on optics, imaging and lighting have played an important role to make a next step toward the sophisticated experiences. Computational photography is one of the most influential technology in computer vision and optical engineering areas, and we think most techniques in computational imaging and projection can be applied to common problems in mixed reality, such as scene modeling, modification of the appearances of actual objects and user interactions.

In this paper we propose a novel depth measurement method by fusing depth from defocus (DFD) and stereo. One of the problems of passive stereo method is the difficulty of finding correct correspondence between images when an object has a repetitive pattern or edges parallel to the epipolar line. On the other hand, the accuracy of DFD method is inherently limited by the effective diameter of the lens. Therefore, we propose the fusion of stereo method and DFD by giving different focus distances for left and right cameras of a stereo camera with coded apertures. Two types of depth cues, defocus and disparity, are naturally integrated by the magnification and phase shift of a single point spread function (PSF) per camera. In this paper we give the proof of the proportional relationship between the diameter of defocus and disparity which makes the calibration easy. We also show the outstanding performance of our method which has both advantages of two depth cues through simulation and actual experiments.

This paper shows a random and distinct shape of each pixel improves the performance of super-resolution using multiple input images. Since the spatial light sensitivity distribution in each pixel of an image sensor is rectangular and identical, the process of imaging is equivalent to the point sampling of blurred image which is a result of convolution of a rectangle with the original image. The convolution results in a loss of the high spatial frequency component of the original image, which limits the performance of super-resolution. Thus, we sprayed a fine-grained black powder on an image sensor to give a random code to the spatial light sensitivity distribution in each pixel. This approach was combined with a reconstruction technique based on sparse regularization, which is commonly used in compressed sensing, in an experiment with an actual setup. A high-resolution image was reconstructed from a limited number of input images and the performance of super-resolution was significantly improved.

Recently it is relatively easy to produce digital point sampled 3D geometric models. In sight of the increasing capability of 3D scanning systems to produce models with millions of points, compression efficiency is of paramount importance. In this paper, we propose a novel competition-based predictive method for single-rate compression of 3D models represented as point cloud. In particular we aim at 3D scanning methods based on grid pattern. The proposed method takes advantage of the pattern characteristic made of vertical and horizontal lines, by assuming that the object surface is sampled in curve of points. We then designed and implemented a predictive coder driven by this curve-based point representation. Novel prediction techniques are specifically designed for a curve-based cloud of points, and been competing between them to achieve high quality 3D reconstruction. Experimental results demonstrate the effectiveness of the proposed method.

Exaggerated defocus can not be created with an ordinary compact digital camera because of its tiny sensor size, so it is hard to take pictures that attract a viewer to the main subject. On the other hand, there are many methods for controlling focus and defocus of previously taken pictures. However, most of these methods require purpose-built equipment such as a camera array to take pictures. Therefore, in this paper, we propose a method to create images focused at any depth with arbitrarily blurred background from the set of images taken by a handheld compact digital camera moved randomly. Using our method, it is possible to produce various aesthetic blurs by changing the size, shape or density of the blur kernel. In addition, we confirm the potential of our method through a subjective evaluation of blurred images created by our system.

We show a new camera based interaction solution where an ordinary camera can detect small optical tags from a relatively large distance. Current optical tags, such as barcodes, must be read within a short range and the codes occupy valuable physical space on products. We present a new low-cost optical design so that the tags can be shrunk to 3mm visible diameter, and unmodified ordinary cameras several meters away can be set up to decode the identity plus the relative distance and angle. The design exploits the bokeh effect of ordinary cameras lenses, which maps rays exiting from an out of focus scene point into a disk like blur on the camera sensor. This bokeh-code or Bokode is a barcode design with a simple lenslet over the pattern. We show that a code with 15 μm features can be read using an off-the-shelf camera from distances of up to 2 meters. We use intelligent binary coding to estimate the relative distance and angle to the camera, and show potential for applications in augmented reality and motion capture. We analyze the constraints and performance of the optical system, and discuss several plausible application scenarios.

In computer vision, background subtraction method is widely used to extract a changing region in a scene. However, it is difficult to simply apply this method to a scene with moving background object, because such object may be extracted as a changing region. Therefore, a method has been proposed to estimate both current background image and occluding object region simultaneously by using eigenspace-based background representation. On the other hand, image completion method using eigenspace have been extended to non-linear subspace using kernel trick, however, such existing method takes large computational cost. Therefore, in this paper, we propose a method for rapid simultaneous estimation of a background image and occluded region in non-linear space, using the kernel trick and iterative projection.

The light transport is one of the useful representation of the optical phenomena inside a scene (Sen et al;, 2005 and Seitz et al., 2005). By using this concept, we can regard any complex scene as a simple linear system between incident light and luminance distribution. However, it is difficult to obtain enough sample under varied lighting condition, because the light source has large degrees of freedom. Therefore, in this paper, we propose a fast and efficient sampling method using characteristics of direct and indirect components. Our method can be divided to three stages: at first, we obtain geometric relationships between projector and camera using Gray code. Then repetive dot patterns are projected to acquire direct reflection component in parallel. Finally, stripe pattern in a small rectangular is projected to model the low-frequency indirect light distribution. The system enables to obtain light transport quickly, and dynamic range of indirect component is also improved. Separated representation of direct and indirect components in light transport allows further applications.

We developed a direction system for assisting the work in a real world from a distant site. At first, the 3-D shape of the object is measured and sent to the distant PC. A supervisor at the distant site can observe the CG of the object and draw annotation figures on it. The figures of the direction message are projected onto the object using projectors. The worker is free from any wearing equipment, ex. HMD, and multi projectors avoid the problem of occlusion by the worker body.

We describe a method for observing multiple targets in a wide-area spatial environment using a distributed vision system (DVS). The DVS is constructed of some 'watching stations' that consist of a camera, an image processor and a computer network that connects each systems. The system's goal is to track multiple people in a wide-area that cannot be watched by single visual sensor. Our approach is based on three algorithms; an algorithm for real-time human tracking, the task decision algorithms of individual watching stations, and the object-matching method used between stations. We also describe experimental results that show the validity of our approach.

In a Cooperative Distributed Vision system a group of communicating Active Vision Agents (AVA, in short, i.e. real time image processor with an active video camera and high speed network interface) cooperate to fulfil a meaningful task such as moving object tracking and dynamic scene visualization. A key issue to design and implement an AVA rests in the dynamic integration of Visual Perception, Camera Action, and Network Communication. This paper proposes a novel dynamic system architecture named Dynamic Memory Architecture, where perception, action, and communication modules share what we call the Dynamic Memory. It maintains not only temporal histories of state variables such as pan-tilt angles of the camera and the target object location but also their predicted values in the future. Perception, action, and communication modules are implemented as parallel processes which dynamically read from and write into the memory according to their own individual dynamics. The dynamic memory supports such asynchronous dynamic interactions (i.e. data exchanges between the modules) without wasting time for synchronization. This no-wait asynchronous module interaction capability greatly facilitates the implementation of real time reactive systems such as moving object tracking. Moreover, the dynamic memory supports the virtual synchronization between multiple AVAs, which facilitates the cooperative object tracking by communicating AVAs. A prototype system for real time moving object tracking demonstrated the effectiveness of the proposed idea.

A range image segmentation contest was organized in conjunction with ICPR'2000. The goal is to continue the effort of experimentally evaluating range image segmentation algorithms initiated by Hoover et al. (1996) and Powell et al. (1998). This paper summarizes the results of the contest.

In this paper, we first introduce the multi-focus camera, a new image sensor used for depth from defocus (DFD) range measurement. It can capture three images with different focus values simultaneously. We then propose two different depth measurement methods using the camera. The first method, an augmented version of the one proposed by N. Asada et al. (1998), employs a noniterative optimization process to compute depth values on edge points. The second one incorporates a coded aperture with the camera; and applies model-based pattern matching to estimate depth values of textured surfaces. Here we propose two types of coded apertures and corresponding analysis algorithms: 1D Fourier analysis to acquire a depth map and a blur-free image from three defocused images taken with a pair of pinholes, and 2D convolution based model matching for the fast and precise depth measurement using a coded aperture with four pinholes. Experimental results showed that the multi-focus camera works well as a practical DFD range sensor and that the coded apertures much improve its range estimation capability for real world scenes.

An application of the pose detection using range images usually uses characteristic matching of the geometrical model but this method has two problems: selecting characteristics from range images is difficult; and it is difficult to make a geometrical model for a complicated shape. Previously a parametric eigen-space method was proposed for pose detection. This method makes object recognition and pose detection possible, but this parametric eigen-space method has a problem that intensity-images depend on a variety of light conditions therefore learning images must include variation of light conditions.

We propose a new method to track 3-D motion of an object in real-time, using range images. A very high speed range sensor "Silicon Range Finder" makes it possible to acquire range images at more than 30 frames per second, and rotated to keep the object in the measurable area. Rough but fast results of range image processing controls the angle of the range sensor, and the object centered range images decide the pose of the object. The 3-D CG rendering hardware of the graphics workstation generates range images from the surface model very fast, so it makes the comparison algorithm simple. Tracking speed and accuracies of estimated motion seem to be feasible for the man-machine cooperative systems.