Some features are specially designed to cope with complex illumination variations [1, 2, 3, 4, 5]. These features are based on intensity orders rather than values so that their descriptors are invariant to non-linear intensity changes. However, one limitation of such features is that they are only able to handle monotonic illumination changes, as the examples shown in Figure 1. In this paper, we try to cope with this problem by proposing a new type of local feature, which describes the context information of neighboring line segments. Through extensive experiments and based on existing research, we found that the best pixel grouping level to handle large illumination variations is using curves or lines. Since curves require more sophisticated groupings and can be approximated by multiple line segments, we use line segments as the primitives of our feature descriptor. We call the proposed feature Line Context, since it is inspired by Shape Context. Line Context Detector. It is well known that edges are present at various scales. To detect edges at different scales we use multi-scale Canny edge detector with Gaussian derivatives at several pre-selected scales. Two phases are used to remove unstable edges. The first phase is to apply Laplacian operator. Those edges that do not attain a distinctive extremum over scales will be removed. The second phase is to remove those edges with homogeneous gradients, i.e. the points where the underlying curves have zero curvatures. We apply Harris matrix to achieve this purpose. Segments in Context. The edge pixels are linked to connected curves at different scales. These curves are then fitted by straight line segments. As shown in Figure 2-(a), several cases need to be considered for representing curves with line segments. One curve may be fitted by multiple segments like curve a. Two segments with small gap in between are merged into one larger segment no matter they are on the same scale (curve b and d) or different scales (curve f and g). In the latter case, the merged segment only exists in the lower scale (segment 8). Besides, all the segments in higher levels are also segments (segment 1, 2 and 3) or part of segments (segment 5 and 7) in lower levels. For each keypoint, we need to find line segments in its neighborhood, which is also called context of the feature. The line segments lying inside or partially inside the context are called context segments. The initial scale σ provides an estimate size of searching area. Let dpoint−set(v,S) be the shortest distance from point v to a set of points S. With keypoint k and all-segments {segi}, the segments set L in the context is defined as,

In this paper, we present an algorithm that is to estimate the position of a hand-held camera with respect to terrestrial LiDAR data. Our input is a set of 3D range scans with intensities and one or a set of 2D uncalibrated camera images of the scene. The algorithm that automatically registers range scans and 2D images is composed of following steps. In the first step, we project the terrestrial LiDAR onto 2D images according to several preselected viewpoints. Intensity-based features such as SIFT are extracted from these projected images and these features are projected back onto the LiDAR data to obtain their 3D positions. In the second step, we estimate the initial pose of given 2D images from feature correspondences. In the third step, we refine the coarse camera pose obtained from the previous step through iterative matchings and optimization process. We presents results from experiments in several different urban settings.

With fast growing popularity of smart phones in recent years, augmented reality (AR) becomes more demanding than ever before. However, one of main challenges is that while features like SIFT or SURF are robust in matchings, they are not computationally efficient. In this paper, we propose an efficient matching method for robust features. A distinctive descriptor is also proposed for performance improvements. Besides, we have developed an outdoor augmented reality system that is based on our proposed methods. The system demonstrates that not only it can achieve robust matchings efficiently, it is also capable to handle large occlusions such as passengers and moving vehicles.

In recent years, interest point based feature such as SIFT and SURF are widely used for image matchings. While these features are robust to changes in scales, rotations, and local geometric deformations, they are usually less able to simultaneously handle viewpoint changes and large illumination changes. In this paper, we will cope with this challenging problem. The basic idea is to relight one of the two images so that the image has similar illumination conditions as the other one. After relighting process, the point based features become effective again in the matching process. Our experiments show that the proposed method has good performance for matching two images with very different illuminations.

With the fast-growing popularity of smart phones in recent years, augmented reality (AR) on mobile devices is gaining more attention and becomes more demanding than ever before. However, the limited processors in mobile devices are not quite promising for AR applications that require real-time processing speed. The challenge exists due to the fact that, while fast features are usually not robust enough in matchings, robust features like SIFT or SURF are not computationally efficient. There is always a tradeoff between robustness and efficiency and it seems that we have to sacrifice one for the other. While this is true for most existing features, researchers have been working on designing new features with both robustness and efficiency. In this article, we are not trying to present a completely new feature. Instead, we propose an efficient matching method for robust features. An adaptive scoring scheme and a more distinctive descriptor are also proposed for performance improvements. Besides, we have developed an outdoor augmented reality system that is based on our proposed methods. The system demonstrates that not only it can achieve robust matchings efficiently, it is also capable to handle large occlusions such as passengers and moving vehicles, which is another challenge for many AR applications.

We present a recognition-based user tracking and augmented reality system that works in extreme large scale areas. The system will provide a user who captures an image of a building facade with precise location of the building and augmented information about the building. While GPS cannot provide information about camera poses, it is needed to aid reducing the searching ranges in image database. A patch-retrieval method is used for efficient computations and real-time camera pose recovery. With the patch matching as the prior information, the whole image matching can be done through propagations in an efficient way so that a more stable camera pose can be generated. Augmented information such as building names and locations are then delivered to the user. The proposed system mainly contains two parts, offline database building and online user tracking. The database is composed of images for different locations of interests. The locations are clustered into groups according to their UTM coordinates. An overlapped clustering method is used to cluster these locations in order to restrict the retrieval range and avoid ping pong effects. For each cluster, a vocabulary tree is built for searching the most similar view. On the tracking part, the rough location of the user is obtained from the GPS and the exact location and camera pose are calculated by querying patches of the captured image. The patch property makes the tracking robust to occlusions and dynamics in the scenes. Moreover, due to the overlapped clusters, the system simulates the "soft handoff" feature and avoid frequent swaps in memory resource. Experiments show that the proposed tracking and augmented reality system is efficient and robust in many cases.

We present a recognition-driven navigation system for large-scale 3D virtual environments. The proposed system contains three parts, virtual environment reconstruction, feature database building and recognition-based navigation. The virtual environment is reconstructed automatically with LIDAR data and aerial images. The feature database is composed of image patches with features and registered location and orientation information. The database images are taken at different distances from the scenes with various viewing angles, and these images are then partitioned into smaller patches. When a user navigates the real world with a handheld camera, the captured image is used to estimate its location and orientation. These location and orientation information are also reflected in the virtual environment. With the proposed patch approach, the recognition is robust to large occlusions and can be done in real time. Experiments show that our proposed navigation system is efficient and well synchronized with real world navigation.

We present a retrieval-based tracking system that requires less computational time and cost. The system tracks a user's location through a small portion of an image captured by the camera, and then refines the camera pose by propagating matchings to the whole image. Augmented information such as building names and locations will be delivered to the user. The progressive way to process image data not only can provide the user with location information at real-time speed, but more importantly, it reduces the feature matching time by limiting the searching ranges. The proposed system contains two parts, offline database building and online user tracking. The database is composed of image patches with features and location information. The images are captured at different locations of interests from different viewing angles and distances, and then these images are partitioned into smaller patches. The location of a user can be calculated by querying one or more patches of the captured image. Moreover, the system is capable to handle large occlusions in images due to the patch approach. Experiments show that the proposed tracking system is efficient and robust in many different environments.

Finding corresponding image points is a challenging computer vision problem, especially for confusing scenes with surfaces of low textures or repeated patterns. Despite the well-known challenges of extracting conceptually meaningful high-level matching primitives, many recent works describe high-level image features such as edge groups, lines and regions, which are more distinctive than traditional local appearance based features, to tackle such difficult scenes. In this paper, we propose a different and more general approach, which treats the image matching problem as a recognition problem of spatially related image patch sets. We construct augmented semi-global descriptors (ordinal codes) based on subsets of scale and orientation invariant local keypoint descriptors. Tied ranking problem of ordinal codes is handled by increasingly keypoint sampling around image patch sets. Finally, similarities of augmented features are measured using Spearman correlation coefficient. Our proposed method is compatible with a large range of existing local image descriptors. Experimental results based on standard benchmark datasets and SURF descriptors have demonstrated its distinctiveness and effectiveness.

We present a robust camera pose estimation approach for stereo images captured in untextured environments. Unlike most of existing registration algorithms which are point-based and make use of intensities of pixels in the neighborhood, our approach imports line segments in registration process. With line segments as primitives, the proposed algorithm is capable to handle untextured images such as scenes captured in man-made environments, as well as the cases when there are large viewpoint changes or illumination changes. Furthermore, since the proposed algorithm is robust to large base-line stereos, there are improvements on the accuracy of 3D points reconstruction. With well-calculated camera pose and object positions in 3D space, we can embed virtual objects into existing scene with higher accuracy for realistic effects. In our experiments, 2D labels are embedded in the 3D scene space to achieve annotation effects as in AR.