Nestor is a real-time recognition and camera pose estimation system for planar shapes. The system allows shapes that carry contextual meanings for humans to be used as Augmented Reality (AR) tracking targets. The user can teach the system new shapes in real time. New shapes can be shown to the system frontally, or they can be automatically rectified according to previously learned shapes. Shapes can be automatically assigned virtual content by classification according to a shape class library. Nestor performs shape recognition by analyzing contour structures and generating projective-invariant signatures from their concavities. The concavities are further used to extract features for pose estimation and tracking. Pose refinement is carried out by minimizing the reprojection error between sample points on each image contour and its library counterpart. Sample points are matched by evolving an active contour in real time. Our experiments show that the system provides stable and accurate registration, and runs at interactive frame rates on a Nokia N95 mobile phone.

Sketching leverages human skills for various purposes. In-Place Augmented Reality Sketching experiences build on the intuitiveness and flexibility of hand sketching for tasks like content creation. In this paper we explore the design space of In-Place Augmented Reality Sketching, with particular attention to content authoring in games. We propose a contextual model that offers a framework for the exploration of this design space by the research community. We describe a sketch-based AR racing game we developed to demonstrate the proposed model. The game is developed on top of our shape recognition and 3D registration library for mobile AR.

We propose a novel approach for text line segmentation based on adaptive local projection profiles. Our algorithm is suitable for degraded documents with text lines written in large skew. The main novelty of our approach is applying the local algorithm in an incremental manner that adapts to the skew of each text line as it progresses. The proposed approach achieves very accurate results on a set of degraded documents with lines written in different skew angles and curvatures.

We present a framework for authoring three-dimensional virtual scenes for Augmented Reality (AR) which is based on hand sketching. Sketches consisting of multiple components are used to construct a 3D virtual scene augmented on top of the real drawing. Model structure and properties can be modified by editing the sketch itself and printed content can be combined with hand sketches to form a single scene. Authoring by sketching opens up new forms of interaction that have not been previously explored in Augmented Reality. To demonstrate the technology, we implemented an application that constructs 3D AR scenes of mechanical systems from freehand sketches, and animates the scenes using a physics engine. We provide examples of scenes composed from trihedral solid models, forces, and springs. Finally, we describe how sketch interaction can be used to author complicated physics experiments in a natural way.

In this paper we present Nestor, a system for real-time recognition and camera pose estimation from planar shapes. The system allows shapes that carry contextual meanings for humans to be used as augmented reality (AR) tracking fiducials. The user can teach the system new shapes at runtime by showing them to the camera. The learned shapes are then maintained by the system in a shape library. Nestor performs shape recognition by analyzing contour structures and generating projective invariant signatures from their concavities. The concavities are further used to extract features for pose estimation and tracking. Pose refinement is carried out by minimizing the reprojection error between sample points on each image contour and its library counterpart. Sample points are matched by evolving an active contour in real time. Our experiments show that the system provides stable and accurate registration, and runs at interactive frame rates on a Nokia N95 mobile phone.

Document skew estimation is an important step in the process of document analysis. In this paper we discuss the properties of the distance transform of binarized documents and derive a fast and accurate method for detecting document skew. The method is based on the observation that the dominant orientation of the gradient of the distance transform accurately reflects the skew of the document. Our experiments suggest the method is robust to large variations in text properties, such as difference in script and page layout, and provides estimation accuracy of state-of-the-art skew detection methods.

In this paper we present a new vision-based approach for transmitting virtual models for augmented reality (AR). A two dimensional representation of the virtual models is embedded in a printed image. We apply image-processing techniques to interpret the printed image and extract the virtual models, which are then overlaid back on the printed image. The main advantages of our approach are: (1) the image of the embedded virtual models and their behaviors are understandable to a human without using an AR system, and (2) no database or network communication is required to retrieve the models. The latter is useful in scenarios with large numbers of users. We implemented an AR system that demonstrates the feasibility of our approach. Applications in education, advertisement, gaming, and other domains can benefit from our approach, since content providers need only to publish the printed content and all virtual information arrives with it.