Airborne LiDAR technology draws increasing interest in large-scale 3D urban modeling in recent years. 3D LiDAR data typically has no texture information. To generate photo-realistic 3D models, oblique aerial images are needed for texture mapping, in which the key step is to obtain accurate registration between aerial images and untextured 3D LiDAR data. We present a robust automatic registration approach. A novel feature called 3CS is proposed which is composed of connected line segments. Putative line segment correspondences are obtained by matching 3CS features detected from both aerial images and 3D LiDAR data. Outliers are removed with a two-level RANSAC algorithm that integrates local and global processing to improve robustness and efficiency. The approach has been tested on 2290 aerial images that cover a variety of urban environments in Oakland and Atlanta areas. Its correct pose recovery rate is over 98%.

We present a wide-baseline image matching approach based on line segments. Line segments are clustered into local groups according to spatial proximity. Each group is treated as a feature called a Line Signature. Similar to local features, line signatures are robust to occlusion, image clutter, and viewpoint changes. The descriptor and similarity measure of line signatures are presented. Under our framework, the feature matching is not only robust against affine distortion but also a considerable range of 3D viewpoint changes for non-planar surfaces. When compared to matching approaches based on existing local features, our method shows improved results with low-texture scenes. Moreover, extensive experiments validate that our method has advantages in matching structured non-planar scenes under large viewpoint changes and illumination variations.

Augmented Virtual Environments (AVE) are very effective in the application of surveillance, in which multiple video streams are projected onto a 3D urban model for better visualization and comprehension of the dynamic scenes. One of the key issues in creating such systems is to estimate the parameters of each camera including the intrinsic parameters and its pose relative to the 3D model. Nowadays, PTZ cameras are popular in this kind of applications. How to rapidly calibrate them at an arbitrary PTZ setting is not clear in the literature. We propose an efficient approach with two steps. In the first step, panoramic images are generated at a set of zooms. The images composing these panoramas are calibrated and stored in a database. In the second step, an image is acquired at an arbitrary PTZ setting. Its best matching image in the database is found by using an efficient local feature recognition technique. Based on this image, the camera parameters at the new PTZ setting can be estimated.

We present a robust camera pose estimation approach for stereo images captured in untextured environments. Unlike most of existing registration algorithms which are point-based and make use of intensities of pixels in the neighborhood, our approach imports line segments in registration process. With line segments as primitives, the proposed algorithm is capable to handle untextured images such as scenes captured in man-made environments, as well as the cases when there are large viewpoint changes or illumination changes. Furthermore, since the proposed algorithm is robust to large base-line stereos, there are improvements on the accuracy of 3D points reconstruction. With well-calculated camera pose and object positions in 3D space, we can embed virtual objects into existing scene with higher accuracy for realistic effects. In our experiments, 2D labels are embedded in the 3D scene space to achieve annotation effects as in AR.

One of the important steps in gradient-based edge detection is thresholding, e.g., hysteresis thresholding in the Canny detector. Traditional approaches only use gradient magnitude as the criterion to select edge pixels. We introduce a novel saliency measure called supporting range. It measures the range along the gradient direction of an edge pixel in which its gradient magnitude is a local maximum. As the result, our approach can detect the edge pixels on object boundaries even if they have very low gradient magnitude. In addition, unlike the Canny detector, the proposed hysteresis thresholding approach is based on segments instead of individual pixels. This also makes the approach more robust to detect edge pixels on weak boundaries.

Aerial imagery and ground-level imagery are two complementary data sources for architectural modeling. How to integrate them is a critical issue in creating complete, photo-realistic and large-scale urban models. We describe a semiautomatic approach of detecting feature correspondences between ground-level images and the building footprint in an orthorectified aerial image. The ground-level images are stitched into panoramas in order to obtain a wide camera field of view. Line segments are extracted from ground-level images. Their corresponding segments on the building footprints are automatically detected through a voting process. Meanwhile the camera pose of the ground-level images is also obtained. Wrong correspondences are corrected through user interaction. Later, the height values of the building roof corners are computed and a piece-wise planar 3D model with photo-realistic facade and roof texture is then created.

Recent years have seen growing interest in mobile augmented reality. The ability to retrieve information and display it as virtual content overlaid on top of an image of the real world is a natural extension to a mobile device equipped with a camera and wireless connectivity. Such applications need to address a number of technical hurdles including target recognition and camera pose estimation. Beyond these fundamental challenges, however, there is the problem of data connectivity and user interface presentation. How do we connect mobile clients to multiple data sources that may be changing in real time and provide the user with a flexible interface for navigating through the relevant content? We present an AR user interface framework specifically designed to expose disparate data sources through a single application server. Our proposed system uses an multi-tier architecture to separate back-end data retrieval from front-end graphical presentation and UI event handling. We describe how it might be used to build an oil platform equipment maintenance system, as an example of a collaborative, data-driven mobile application.

Augmented virtual environments (AVE) are very effective in the application of surveillance, in which multiple video streams are projected onto a 3D urban model for better visualization and comprehension of the dynamic scenes. One of the key issues in creating such systems is to estimate the parameters of each camera including the intrinsic parameters and its pose relative to the 3D model. Existing camera pose estimation approaches require known intrinsic parameters and at least three 2D to 3D feature (point or line) correspondences. This cannot always be satisfied in an AVE system. Moreover, due to noise, the estimated camera location may be far from the expectation of the users when the number of correspondences is small. Our approach combines the users' prior knowledge about the camera location and the constraints from the parallel relationship between lines with those from feature correspondences. With at least two feature correspondences, it can always output an estimation of the camera parameters that gives an accurate alignment between the projection of the image (or video) and the 3D model