Augmented reality (AR) marker hiding is a technique to visually remove AR markers in a real-time video stream. A conventional approach transforms a background image with a homography matrix calculated on the basis of a camera pose and overlays the transformed image on an AR marker region in a real-time frame, assuming that the AR marker is on a planar surface. However, this approach may cause discontinuities in textures around the boundary between the marker and its surrounding area when the planar surface assumption is not satisfied. This paper proposes a method for AR marker hiding without discontinuities around texture boundaries even under nonplanar background geometry without measuring it. For doing this, our method estimates the dense motion in the marker's background by analyzing the motion of sparse feature points around it, together with a smooth motion assumption, and deforms the background image according to it. Our experiments demonstrate the effectiveness of the proposed method in various environments with different background geometries and textures.

3D models of outdoor environments have been used for several applications such as a virtual earth system and a vision-based vehicle safety system. 3D data for constructing such 3D models are often measured by an on-vehicle system equipped with laser rangefinders, cameras, and GPS/IMU. However, 3D data of moving objects on streets lead to inaccurate 3D models when modeling outdoor environments. To solve this problem, this paper proposes a moving object detection method for point clouds by minimizing an energy function based on photometric and depth consistencies assuming that input data consist of synchronized point clouds, images, and camera poses from a single sequence captured with a moving on-vehicle system.

Diminished reality aims to remove real objects from video images and fill in the missing regions with plausible background textures in real time. Most conventional methods based on image inpainting achieve diminished reality by assuming that the background around a target object is almost planar. This paper proposes a new diminished reality method that considers background geometries with less constraints than the conventional ones. In this study, we approximate the background geometry by combining local planes, and improve the quality of image inpainting by correcting the perspective distortion of texture and limiting the search area for finding similar textures as exemplars. The temporal coherence of texture is preserved using the geometries and camera pose estimated by visual-simultaneous localization and mapping (SLAM). The mask region that includes a target object is robustly set in each frame by projecting a 3D region, rather than tracking the object in 2D image space. The effectiveness of the proposed method is successfully demonstrated using several experimental environments.

This paper proposes an example-based method to increase the resolution of a low-resolution image. In the proposed method, we generate example images by a novel view synthesis technique using 3D geometry reconstruction and camera pose estimation from a video or images capturing the same scene. We then increase the resolution by minimizing an energy function by searching for the optimal example from the generated example images. The proposed method has less limitations on camera positions and geometry of the target scene than those in conventional methods. Experiments demonstrate the effectiveness of the proposed method by qualitatively comparing the results of the proposed and conventional methods.

This paper proposes a new diminished reality method for 3D scenes considering background structures. Most conventional methods using image inpainting assumes that the background around a target object is almost planar. In this study, approximating the background structure by the combination of local planes, perspective distortion of texture is corrected and searching area is limited for improving the quality of image inpainting. The temporal coherence of texture is preserved using the estimated structures and camera pose estimated by Visual-SLAM.

Augmented reality (AR) images with virtualized real objects can be used for various applications. However, such AR image generation requires hand-crafted 3D models of that objects, which are usually not available. This paper proposes a view-dependent texture (VDT)- and view-dependent geometry (VDG)-based method for generating high quality AR images, which uses 3D models automatically reconstructed from multiple images. Since the quality of reconstructed 3D models is usually insufficient, the proposed method inflates the objects in the depth map as VDG to repair chipped object boundaries and assigns a color to each pixel based on VDT to reproduce the detail of the objects. Background pixel exposure due to inflation is suppressed by the use of the foreground region extracted from the input images. Our experimental results have demonstrated that the proposed method can successfully reduce above visual artifacts.

ISMAR 2012: IEEE International Symposium on Mixed and Augmented Reality , Nov 5-8, 2012 , Atlanta, GA, USA

In this paper, we propose a novel surface completion method to generate plausible shapes and textures for missing regions of 3D models. The missing regions are filled in by minimizing two energy functions for shape and texture, which are both based on similarities between the missing region and the rest of the object; in doing so, we take into account the positive correlation between shape and texture. We demonstrate the effectiveness of the proposed method experimentally by applying it to two models.

Surface completion is a technique for filling missing regions in 3D models measured by range scanners and videos. Conventionally, although missing regions were filled with the similar shape in a model, the completion process was fairly inefficient because the whole region in the model was searched for the similar shape. In this paper, the completion is efficiently performed using principal curvatures of local shape. In experiments, the effectiveness of the proposed method is successfully verified with subjective evaluation. In addition, the quantitative evaluation which has not been in the literature is newly performed.

3D mesh models generated with range scanner or video images often have holes due to many occlusions by other objects and the object itself. This paper proposes a novel method to fill the missing parts in the incomplete models. The missing parts are filled by minimizing the energy function, which is defined based on similarity of local shape between the missing region and the rest of the object. The proposed method can generate complex and consistent shapes in the missing region. In the experiment, the effectiveness of the method is successfully demonstrated by applying it to complex shape objects with missing parts.