We propose a method that estimates 6-DoF camera pose from a partially visible large object, by exploiting information of its subparts that are detected using a state-of-the-art convolutional neural network (CNN). The trained CNN outputs two-dimensional bounding boxes around subparts and associated classes. Information from detection is then fed to a deep neural network that regresses to camera's 6-DoF poses. Experimental results show that the proposed method is more robust to occlusions than conventional learning-based methods.

Imagine that one may have access to a part of a text document, say a page, and from that would want to identify the document to which it belongs. In such cases, there is a need to perform a content-based document retrieval in a large database.

Edge-based tracking is a fast and plausible approach for textureless 3D object tracking, but its robustness is still very challenging in highly cluttered backgrounds due to numerous local minima. To overcome this problem, we propose a novel method for fast and robust textureless 3D object tracking in highly cluttered backgrounds. The proposed method is based on optimal local searching of 3D-2D correspondences between a known 3D object model and 2D scene edges in an image with heavy background clutter. In our searching scheme, searching regions are partitioned into three levels (interior, contour, and exterior) with respect to the previous object region, and confident searching directions are determined by evaluating candidates of correspondences on their region levels; thus, the correspondences are searched among likely candidates in only the confident directions instead of searching through all candidates. To ensure the confident searching direction, we also adopt the region appearance, which is efficiently modeled on a newly defined local space (called a searching bundle). Experimental results and performance evaluations demonstrate that our method fully supports fast and robust textureless 3D object tracking even in highly cluttered backgrounds.

Hand shape is a natural and human-friendly interface for human-computer interaction. This paper proposes a real- time and 2D vision-based hand shape recognition method. The method is robust to hand pose changes because the hand pose is neutralized after recognizing a hand pose using distance transform, principal component analysis (PCA), and histogram analysis. Also, the context-based recognition method using shape decomposition can effectively recognize tiny changes of fingers. The method worked at 44.8 fps and had a recognition rate of 83% on average in the experiment with 800 images including 5 hand shapes and 16 hand poses.

This paper proposes an augmented reality interface that provides natural hand-based interaction with virtual objects on mobile phones. Assume that one holds a mobile phone in a hand and sees the other hand through mobile phone's camera. Then, a virtual object is rendered on his/her palm and reacts to hand and finger movements. Since the proposed interface does not require any additional sensors or markers, one freely interacts with the virtual object anytime and anywhere. The proposed interface worked at 5 fps on a mobile phone (Galaxy S2 having a dual-core processor).

Human, especially human-of-interest, detection and tracking is a necessary part for immersive and realistic human-computer interaction. Thus, we propose an intelligent multi-vision-based human detection and tracking system that consists of a visible camera, an infrared camera, a dynamic infrared lighting system, and a cold mirror. The system segments a human region of interest and tracks his/her motion. Experimental results are given for verifying the usefulness of the proposed method.

To estimate camera pose, existing augmented reality (AR) techniques usually require fiducial markers with known geometry. However, placing a marker in the workspace of the user can be very visually intrusive. To overcome this limitation, a nonintrusive AR method using invisible markers that are created/drawn with an infrared (IR) ink is proposed and an IR marker tracking system is presented. This system includes additional algorithms to maintain reliable performance in a cluttered background. Then, the working conditions of IR markers are examined and compared with those of visual markers, and the optimal working conditions of IR markers are discussed. Next, the qualitative evaluation of an IR marker–based AR is presented through a variety of experiments and user evaluations. Finally, the potential applications of IR marker–based AR are explored.

Abstract Model‐based camera tracking is a technology that estimates a precise camera pose based on visual cues (e.g., feature points, edges) extracted from camera images given a 3D scene model and a rough camera pose. This paper proposes an automatic method for flexibly adjusting the confidence of visual cues in model‐based camera tracking. The adjustment is based on the conditions of the target object/scene and the reliability of the initial or previous camera pose. Under uncontrolled or less‐controlled working environments, the proposed object‐adaptive tracking method works flexibly at 20 frames per second on an ultra mobile personal computer (UMPC) with an average tracking error within 3 pixels when the camera image resolution is 320 by 240 pixels. This capability enabled the proposed method to be successfully applied to a mobile augmented reality (AR) guidance system for a museum. Copyright © 2009 John Wiley & Sons, Ltd.

In projection-based augmented reality (AR) alleviating visual distraction of patterns has been a great challenge. As a representative one, a method of embedding patterns and their complements (hereafter, we call the pairs complementary patterns) into AR images alternately has been proposed recently. This paper presents subjective evaluation results and their statistical analysis on the visual perceptibility of embedding complementary patterns in different ways in a standard hardware environment. Then, we explore the constraints for embedded complementary patterns to be less perceptible. As expected, high projector refresh rate and low pattern strength were the general conditions for a decrease in the perception of embedded complementary patterns. However, reducing pattern size and projecting complementary patterns with an interval were also among the factors affecting the results. Detailed constraints are given in the experimental results. Also, we present which constraint is more dominant for pattern perceptibility.

We propose a nonintrusive method for generating all-focused projection. Unlike traditional restoration algorithms that operate on a blurred image to recover the original, the nature of our problem requires that the correction be applied to the original image before blurring. To accomplish this, a pattern image is generally projected and a series of spatially varying point-spread-functions (PSFs) across the projection input image were estimated from its camera image. In this paper, the pattern image was imperceptibly projected and thus not visible to the observers. Experimental results demonstrate that the proposed method effectively removes the out-of-focus projection blur in dynamic environments where the projector or screen object moves.

Existing geometric and radiometric compensation methods for direct-projected augmented reality focus on static projection surfaces rather than dynamic surfaces (with varying geometry in time). We aim at providing an effective framework for projecting a sequence of augmented reality images onto dynamic surfaces without geometric and radiometric distortion. We present our design of a special pattern image for simultaneous geometric and radiometric compensation and evaluate two different techniques for embedding the pattern image into augmented reality images. The validity of the proposed method is examined through a variety of experiments with a mobile projector-camera system.

In direct-projected augmented reality, use of projector makes it possible to utilize 3-D real and large objects as displays and frees from discomforts incidental to wearing a device such as HMD. However, the resulting augmentation usually has poor depth resolution due to projectors with low contrast. In direct-projected augmented reality, the radiometric compensation, which is originally employed to recover the color properties of the projector input image in direct-projected augmented reality, seems to be another cause of decreasing the final contrast of the augmentation. In this paper, a contrast enhancement method is proposed that combines a hue- and saturation-preserving histogram equalization with the radiometric compensation in direct-projected augmented reality. The method guarantees that the radiometrically compensated color is maintained while the brightness (or intensity) contrast is enhanced. Experimental results demonstrate the validity of our method

In projection-based augmented reality, specular reflection may distract the users. This work demonstrates that the specular reflection can be eliminated by redundantly illuminating the projection surface using multiple overlapping projectors mounted at different locations. Our initial system using two projectors is presented. The system automatically determines the pixels which include specular reflection in one projector, blank the light falling on the pixel and boost the other projector's output so that the added light is projected onto the pixel consistently.

We introduce a novel tracking system based on invisible markers which are created/drawn with an IR fluorescent pen. The tracking system consists of one scene camera, one IR camera, and one half mirror. The two cameras are positioned in each side of half mirror so that their optical centers coincide with each other. We track the invisible markers using the IR camera and visualize AR in the view of the scene camera. Thus, it works as a robust marker-less tracking system. Experimental results are given to demonstrate the viability of the proposed system.