Given a set of 3D to 2D putative matches, labeling the correspondences as inliers or outliers plays a critical role in a wide range of computer vision applications including the Perspective-n-Point (PnP) and object recognition. In this paper, we study a more generalized problem which allows the matches to belong to multiple objects with distinct poses. We propose a deep architecture to simultaneously label the correspondences as inliers or outliers and classify the inliers into multiple objects. Specifically, we discretize the 3D rotation space into twenty convex cones based on the facets of a regular icosahedron. For each facet, a facet classifier is trained to predict the probability of a correspondence being an inlier for a pose whose rotation normal vector points towards this facet. An efficient RANSAC-based post-processing algorithm is also proposed to further process the prediction results and detect the objects. Experiments demonstrate that our method is very efficient compared to existing methods and is capable of simultaneously labeling and classifying the inliers of multiple objects with high precision.

This research investigates enhancing first-person view (FPV) task instruction videos by applying Augmented Reality (AR) visualisation of spatial cues. With personal mobile devices, recording and sharing a video clip has become very easy, and how-to videos are becoming popular on social video sharing services. Instructional videos are actively used not only in formal education and training, but also in everyday life. However, video clips are limited to two-dimensional representation of the task space, making it hard for the viewer to follow and match the objects in the video to those in the real world task space. We propose augmenting task instruction videos with AR visualisation of spatial cues to overcome this problem, focusing on creating and viewing FPV instruction videos. We designed and implemented a prototype system, AR Tips, which allows users to capture and share augmented FPV instruction videos on a wearable AR device. We conducted a user study to evaluate the benefit of our approach, and the results showed that with the help of augmented spatial cues users better understood the instructions, performed the tasks faster with fewer errors, and had lower mental effort.

Recognizing three dimensional chess pieces using computer vision is needed for an augmented reality chess assistant. This paper proposes an efficient 3D pieces recognition approach based on oriented chamfer matching. During a real game, the pieces might be occluded by other pieces and have varying rotation and scales with respect to the camera. Furthermore, different pieces share lots of similar texture features which makes them more difficult to identify. Our approach addresses the above problems and is capable of identifying the pieces with different scales, rotation and viewing angles. After marking the possible chessboard squares that contain pieces, the oriented chamfer scores are calculated for alternative templates and the recognized pieces are indicated on the input image accordingly. Our approach shows high recognition accuracy and efficiency in experiments and the recognition process can be easily generalized to other pattern recognition applications with 3D templates. Our approach outperforms the convolutional neural networks under severe occlusion and low resolution conditions and has comparative processing time while avoids the time consuming training process.

Chessboards are commonly used to calibrate cameras, and many robust methods have been developed to recognize the unoccupied boards. However, when the chessboard is populated with chess pieces, such as during an actual game, the problem of recognizing the board is much harder. Challenges include occlusion caused by the chess pieces, the presence of outlier lines and low viewing angles of the chessboard. In this paper, we present a novel approach to address the above challenges and recognize the chessboard. The Canny edge detector and Hough transform are used to capture all possible lines in the scene. The k-means clustering and a k-nearest-neighbors inspired algorithm are applied to cluster and reject the outlier lines based on their Euclidean distances to the nearest neighbors in a scaled Hough transform space. Finally, based on prior knowledge of the chessboard structure, a geometric constraint is used to find the correspondences between image lines and the lines on the chessboard through the homography transformation. The proposed algorithm works for a wide range of the operating angles and achieves high accuracy in experiments.

We present an algorithm that can segment and track partial planar templates, from a sequence of images taken from a moving camera. By “partial planar template”, we mean that the template is the projection of a surface patch that is only partially planar; some of the points may correspond to other surfaces. The algorithm segments each image template to identify the pixels that belong to the dominant plane, and determines the three dimensional structure of that plane. We show that our algorithm can track such patches over a larger visual angle, compared to algorithms that assume that patches arise from a single planar surface. The new tracking algorithm is expected to improve the accuracy of visual simultaneous localization and mapping, especially in outdoor natural scenes where planar features are rare.

Pedestrian detection in low resolution videos can be challenging. In outdoor surveillance scenarios, the size of pedestrians in the images is often very small (around 20 pixels tall). The most common and successful approaches for single frame pedestrian detection use gradient-based features and a support vector machine classifier. We propose an extension of these ideas, and develop a new algorithm that extracts gradient features from a spatiotemporal volume, consisting of a short sequence of images (about one second in duration). The additional information provided by the motion of the person compensates for the loss of resolution. On standard datasets (PETS2001, VIRAT) we show a significant improvement in performance over single-frame detection.

In this paper we propose a new data fitting method which, similar to RANSAC, fits data to a model using sample and consensus. The application of interest is fitting 3D point clouds to a prior geometric model. Where the RANSAC process uses random samples of points in the fitting trials, we propose a novel method which directs the sampling by ordering the points according to their contribution to the solution's constraints. This is particularly important when the data is quasi-degenerate. In this case, the standard RANSAC algorithm often fails to find the correct solution. Our approach selects points based on a Mutual Information criterion, which allows us to avoid redundant points that result in degenerate sample sets. We demonstrate our approach on simulated and real data and show that in the case of quasi-degenerate data, the proposed algorithm significantly outperforms RANSAC.

Object pose may be determined from a set of 2D image points and corresponding 3D model points, given the camerapsilas intrinsic parameters. In this paper, two new exterior orientation algorithms are proposed and then compared against the Efficient PnP method and orthogonal iteration algorithm. As an alternative to the homogeneous transformation, both algorithms utilize the 3-simplex as a pose parameterization. One algorithm uses a semidefinite program and the other a Gauss-Newton algorithm.

A new vision and inertial pose estimation system was implemented for real-time handheld augmented reality (AR). A sparse set of 3D cone fiducials are utilized for scalable indoor/outdoor tracking, as opposed to traditional planar patterns. The cones are easy to segment and have a large working volume which makes them more suitable for many applications. The pose estimation system receives measurements from the camera and IMU at 30 Hz and 100 Hz respectively. With a dual-core workstation, all measurements can be processed in real-time to update the pose of virtual graphics within the AR display.

New methods are presented to model, visualize and automatically recognize wide-area activities, which essentially are activities that span large areas (such as a facility or urban neighborhood) and that usually span long time intervals (such as hours and weeks). We introduce the no-go topology method and the chokepoint-observation interaction method, and then show how new algorithms can be built on them to recognize a category of wide-area activity, called process-type activities. Experimental results are presented for recognizing a manufacturing process observed via persistent GMTI sensor data. Then we present experimental results illustrating how an interesting activity can be detected as a deviation from a learned wide-area normalcy model, and how new widearea activity patterns can be discovered using simple visualizations of the results. One objective of this paper is to demonstrate that it is theoretically possible to recognize wide-area and process-type activities in built-up environments using GMTI data. The results presented here use somewhat ideal sensor data (small positional error ellipses, continuous GMTI observations, repetitive activities) and our approach is to move toward more operationally realistic data.

This paper addresses the problem of controlling a mobile robot using a small, portable user interface. In some applications, including security and military, a user might command a mobile robot in the field using a wearable computer. The wearable computer may have a small display that would make information delivery difficult. Also, full video feedback from the robot may not be available due to radio bandwidth limitations. One solution is to have the user interact with a virtual model of the robot environment (available from other sources). We designed an interface to interact with such a model, using a small display. The user controls the robot by gesturing with a single hand. The gestures are sensed using an instrumented glove and a 6 DOF (degree of freedom) magnetic tracker mounted on the user's hand. The interface was tested using two different tasks: a 6 DOF manipulation task and a visualization task which was primarily 3 DOF. For comparison, a simple, 2 DOF input device was also implemented. It was found that the 6 DOF input device was clearly superior for the manipulation task, but results were inconclusive for the visualization task.

A method was developed for registering three-dimensional knee implant models to single plane X-ray fluoroscopy images. We use a direct image-to-image similarity measure, taking advantage of the speed of modern computer graphics workstations to quickly render simulated (predicted) images. As a result, the method does not require an accurate segmentation of the implant silhouette in the image (which can be prone to errors). A robust optimization algorithm (simulated annealing) is used that can escape local minima and find the global minimum (true solution). Although we focus on the analysis of total knee arthroplasty (TKA) in this paper, the method can be (and has been) applied to other implanted joints, including, but not limited to, hips, ankles, and temporomandibular joints. Convergence tests on an in vivo image show that the registration method can reliably find poses that are very close to the optimal (i.e., within 0.4 degrees and 0.1 mm), even from starting poses with large initial errors. However, the precision of translation measurement in the Z (out-of-plane) direction is not as good. We also show that the method is robust with respect to image noise and occlusions. However, a small amount of user supervision and intervention is necessary to detect cases when the optimization algorithm falls into a local minimum. Intervention is required less than 5% of the time when the initial starting pose is reasonably close to the correct answer, but up to 50% of the time when the initial starting pose is far away. Finally, extensive evaluations were performed on cadaver images to determine accuracy of relative pose measurement. Comparing against data derived from an optical sensor as a "gold standard," the overall root-mean-square error of the registration method was approximately 1.5 degrees and 0.65 mm (although Z translation error was higher). However, uncertainty in the optical sensor data may account for a large part of the observed error.

This paper describes a method for tracking human body motion from multiple views in real-time. The method extracts silhouettes in each view using background subtraction, and then intersects the visual hulls generated by the silhouettes to create a set Of voxels. The voxels then exert attractive forces on a kinematic model of the human body to align the model with the voxels. The linked nature of the model allows tracking of partially occluded limbs. The size parameters of the kinematic model are determined automatically during an initialization phase. The kinematic model also incorporates velocity, joint angle, and self collision limits. The entire system with four cameras runs on a single PC in real-time at 20 frames per second. Experiments are presented comparing the performance of the system on real and synthetic imagery to ground truth data.

The need to register data is abundant in applications such as: world modeling, part inspection and manufacturing, object recognition, pose estimation, robotic navigation, and reverse engineering. Registration occurs by aligning the regions that are common to multiple images. The largest difficulty in performing this registration is dealing with outliers and local minima while remaining efficient. A commonly used technique, iterative closest point, is efficient but is unable to deal with outliers or avoid local minima. Another commonly used optimization algorithm, simulated annealing, is effective at dealing with local minima but is very slow. Therefore, the algorithm developed in this paper is a hybrid algorithm that combines the speed of iterative closest point with the robustness of simulated annealing. Additionally, a robust error function is incorporated to deal with outliers. This algorithm is incorporated into a complete modeling system that inputs two sets of range data, registers the sets, and outputs a composite model.

A method is developed to analyze the accuracy of the relative head-to-object position and orientation (pose) in augmented reality systems with head-mounted displays. From probabilistic estimates of the errors in optical tracking sensors, the uncertainty in head-to-object pose can be computed in the form of a covariance matrix. The positional uncertainty can be visualized as a 3D ellipsoid. One useful benefit of having an explicit representation of uncertainty is that we can fuse sensor data from a combination of fixed and head-mounted sensors in order to improve the overall registration accuracy. The method was applied to the analysis of an experimental augmented reality system, incorporating an optical see-through head-mounted display, a head-mounted CCD camera, and a fixed optical tracking sensor. The uncertainty of the pose of a movable object with respect to the head-mounted display was analyzed. By using both fixed and head mounted sensors, we produced a pose estimate that is significantly more accurate than that produced by either sensor acting alone.

Sensing to recognize and locate objects is a critical need for robotic operations in unstructured environments. An accurate 3D model of objects in the scene is necessary for efficient high level control of robots. Drawing upon concepts from supervisory control, we have developed an interactive system for creating object models from range data, based on simulated annealing. Site modeling is a task that is typically performed using purely manual or autonomous techniques, each of which has inherent strengths and weaknesses. However, an interactive modeling system combines the advantages of both manual and autonomous methods, to create a system that has high operator productivity as well as high flexibility and robustness. Our system is unique in that it can work with very sparse range data, tolerate occlusions, and tolerate cluttered scenes. We have performed an informal evaluation with four operators on 16 different scenes, and have shown that the interactive system is superior to either manual or automatic methods in terms of task time and accuracy.

The paper describes an algorithm to estimate the position and orientation (pose) of artificial knee implants from X-ray fluoroscopy images using computer vision. The resulting information is used to determine the kinematics of bone motion in implanted knees. This determination can be used to support the development of improved prosthetic knee implants, which currently have a limited life span due to premature wear of the polyethylene material at the joint surface. The algorithm determines the full 6 degree of freedom translation and rotation of knee components. This is necessary for artificial knees which have shown significant rotation out of the sagittal plane, in particular internal/external rotations. By creating a library of images of components at known orientation and performing a template matching technique, the 3D pose of the femoral and tibial components are determined. The entire process, when used at certain knee angles, gives a representation of the positions in contact during normal knee motion.

An algorithm that uses optical flow to detect landing hazards for a descending spacecraft is described. Image edge points are tracked between frames of a motion sequence, and the range to the points is calculated from the displacement of the edge points and the known motion of the camera. A novel variable-sized edge detector is used to compensate for the change in distance from one image to the next. Kalman filtering is used to incrementally improve the range estimates to those points and provide an estimate of the uncertainty in each range. Errors in camera motion and image point measurement are also modeled. A surface is then interpolated to these points, providing a complete map from which hazards such as steeply sloping areas can be detected. The algorithm has been applied to synthetic and real image sequences, with resulting range accuracy on the order of 1-3% of the range.>

An approach is described that integrates the processes of feature matching, contour detection, and surface interpolation to determine the three-dimensional distance, or depth, of objects from a stereo pair of images. Integration is necessary to ensure that the detected surfaces are smooth. Surface interpolation takes into account detected occluding and ridge contours in the scene; interpolation is performed within regions enclosed by these contours. Planar and quadratic patches are used as local models of the surface. Occluded regions in the image are identified, and are not used for matching and interpolation. A coarse-to-fine algorithm is presented that generates a multiresolution hierarchy of surface maps, one at each level of resolution. Experimental results are given for a variety of stereo images.>