This paper presents a robust but simple object pose tracking algorithm based on Kalman filtering. Compared to markerless pose tracking, a fiducial marker called ArUco provides a fast and accurate solution to the problem. With these advantages, this marker-based technique is ready to be used in virtual reality and operate in low-cost wearable devices. However, it still suffers from the problem of occlusion and noise. If a large part of the marker is occluded, no pose information can be acquired for the moment. Noises due to hand shaking also affect the quality of the resulting pose. This is not desirable in a real-time environment. We tackle the problems by employing a linear Kalman filter. The pose information can be estimated even if the camera view is blocked temporarily. We have performed real experiments to demonstrate the effects of the application of Kalman filter. The results are satisfactory.

Vision-based object tracking is a challenging problem. In the tracking process, the object is usually first recognized in a given image. Then a bounding box is used to describe the position of the target object. Normally, a vector [x, y, width, height] is adopted to represent the bounding box. Under this viewpoint, the tracking problem can be treated as a regression problem if we handle the image sequence frame by frame. Due to the recent advancement in machine learning, many researchers apply neural networks to solve the visual tracking problem. This greatly improves the accuracy of bounding box prediction. Actually, the neural network based approaches are more suitable for end-to-end systems. In this paper, we propose to train and use a single neural network to tackle the tracking task. With the cropped candidate image patch as the input to the network, the output is the bounding box that indicates the target position. In our network, we first have a mask map to identify the target. It is a binary image and is divided into two classes. The positive class denotes the foreground while the negative class denotes the background. The mask map is then used for the estimation of the bounding box vector. The task now becomes an image mapping problem. We have achieved a good balance between accuracy and computational efficiency. Our tracker can reach an average speed of 178 frames per second(fps) and a maximum of 334 fps in the OTB benchmark.

Estimating the 3-D motion of a moving camera from images is a common task in robotics and augmented reality. Most existing marker-less approaches make use of either points or lines. Taking the advantages of both kinds of features in an unknown environment is more attractive due to their availability and differences in characteristics. A novel model-less method is presented in this paper to tackle the 3-D motion tracking problem. Two Bayesian filters, one for point measurements while another for line measurements, are embedded in the Interacting Probabilistic Switching (IPS) framework. They compensate for the weaknesses in one another by utilizing both kinds of features in the stereo images. The proposed method is able to obtain the 3D motion given as little as two line or two point correspondences in consecutive images with the use of multiple trifocal tensors. Our method outperformed two recent methods in terms of accuracy and the problem of drifting was very little in real scenarios.

In this paper, we propose a robust method for visual tracking relying on mean shift, sparse coding and spatial pyramids. Firstly, we extend the original mean shift approach to handle orientation space and scale space and name this new method as mean transform. The mean transform method estimates the motion, including the location, orientation and scale, of the interested object window simultaneously and effectively. Secondly, a pixel-wise dense patch sampling technique and a region-wise trivial template designing scheme are introduced which enable our approach to run very accurately and efficiently. In addition, instead of using either holistic representation or local representation only, we apply spatial pyramids by combining these two representations into our approach to deal with partial occlusion problems robustly. Observed from the experimental results, our approach outperforms state-of-the-art methods in many benchmark sequences.

In this paper, we propose a novel geometric approach for solving camera calibration efficiently. Our camera calibration procedure is implemented on planar objects with rectangles on them. Images of the planar object are taken from different views where corner features are extracted. The geometric relationship between the rectangle corners is utilized to reduce the number of extrinsic unknown parameters. This process can further reduce the complexity of the optimization space greatly and thus enable our approach to be more accurate and more efficient than other geometric approaches. Experimental results on synthetic images and real images witness the advantage of our approach over state-of-the-art approaches.

Keystone correction is an essential operation for projector-based applications, especially in mobile scenarios. In this paper, we propose a handheld movable projection method that can freely project keystone-free content on a general flat surface without adding any markings or boundary on it. Such a projection system can give the user greater freedom of display control (such as viewing angle, distance, etc.), without suffering from keystone distortion. To achieve this, we attach a camera to the projector to form a camera-projector pair. A green frame with the same resolution as the projector screen is projected onto the screen. Particle filter is employed to track the green frame and the correction of the display content is then achieved by rectifying the projection region of interest into a rectangular area. We built a prototype system to validate the effectiveness of the method. Experimental results show that our method can continuously project distortion free content in real time with good performance.

In this paper, we propose a hand-held mobile projection system that can freely project keystone free content on a general flat surface without any markings. Such a projection system can give the user great freedom of control of the display such as viewing angle and distance without suffering from distortion. We attach a camera to the projector to form a stereo pair. The correction of the display content is then achieved by rectifying the projection region of interest into a rectangular region and pre-warping the display content. Experimental results show that our system can continuously project distortion free content in real time with reasonable precision.

In this paper, we proposed a movable hand-held display system which uses a projector to project display content onto an ordinary cardboard which can move freely within the projection area. Such a system can give users greater freedom of control of the display such as the viewing angle and distance. At the same time, the size of the cardboard can be made to a size that fits one's application. A projector-camera pair is calibrated and used as the tracking and projection system. We present a vision based algorithm to detect an ordinary cardboard and track its subsequent motion. Display content is then pre-warped and projected onto the cardboard at the correct position. Experimental results show that our system can project onto the cardboard in reasonable precision.

This paper presents a novel method that acquires camera position and orientation from a stereo image sequence without prior knowledge of the scene. To make the algorithm robust, the interacting multiple model probabilistic data association filter (IMMPDAF) is introduced. The interacting multiple model (IMM) technique allows the existence of more than one dynamic system in the filtering process and in return leads to improved accuracy and stability even under abrupt motion changes. The probabilistic data association (PDA) framework makes the automatic selection of measurement sets possible, resulting in enhanced robustness to occlusions and moving objects. In addition to the IMMPDAF, the trifocal tensor is employed in the computation so that the step of reconstructing the 3-D models can be eliminated. This further guarantees the precision of estimation and computation efficiency. Real stereo image sequences have been used to test the proposed method in the experiment. The recovered 3-D motions are accurate in comparison with the ground truth data and have been applied to control cameras in a virtual environment.

Multiple Camera Systems (MCS) have been widely used in many vision applications and attracted much attention recently. There are two principle types of MCS, one is the Rigid Multiple Camera System (RMCS); the other is the Articulated Camera System (ACS). In a RMCS, the relative poses (relative 3-D position and orientation) between the cameras are invariant. While, in an ACS, the cameras are articulated through movable joints, the relative pose between them may change. Therefore, through calibration of an ACS we want to find not only the relative poses between the cameras but also the positions of the joints in the ACS. Although calibration methods for RMCS have been extensively developed during the past decades, the studies of ACS calibration are still rare. In this paper, two ACS calibration methods are proposed. The first one uses the feature correspondences between the cameras in the ACS. The second one requires only the ego-motion information of the cameras and can be used for the calibration of the non-overlapping view ACS. In both methods, the ACS is assumed to have performed general transformations in a static environment. The efficiency and robustness of the proposed methods are tested by simulation and real experiments. In the real experiment, the intrinsic and extrinsic parameters of the ACS are calibrated using the same image sequences, no extra data capturing step is required. The corresponding trajectory is recovered and illustrated using the calibration results of the ACS. To our knowledge, we are the first to study the calibration of ACS.

The extended Kalman filter (EKF) is suitable for real-time pose estimation due its low computational demand and ability to handle the nonlinear perspective camera model. There are many EKF based approaches in the literature; some are very recent while others exist for about two decades. These methods differ in two main aspects: the number and arrangement of cameras, and the number and usage of filters. In this work, we will compare these approaches using simulations and real experiments. As far as we know, it is the first attempt to do this with such details. We will show which is suitable under different motion patterns, and explain the effect of the bas-relief ambiguity upon the accuracy of the different approaches. Additionally, we will discuss how to solve the scale factor ambiguity, and suggest the best strategy to deal with the features fed to the filter.

Processing of stereo images has become more and more important in recent years because of the availability of various stereo displaying devices. In particular, stabilizing of stereo images is important and useful especially when the images are obtained from cameras held by inexperienced hands or placed on unstable platforms. In this paper, we propose a new frame warping method for such a problem. Most video stabilization methods to date use 2-D geometric transform to approximate the changes between frames. However, these methods fail when there is a large depth variation between the foreground and the background in the scene. We try to solve this by estimating the 3-D motion parameters of the cameras by tri-focal tensor and the extended Kalman filter. And use the motion parameters to stabilize the images. We test the method using synthetic and real images and the results show that the performance of our proposed method is accurate even if the background contains large relative depth variations.

The novelty of this paper is the introduction of the interacting multiple model probabilistic data association filter (IMMPDAF) to the pose tracking problem. The interacting multiple model (IMM) technique allows the existence of more than one dynamic system in the filtering process and in return leads to improved accuracy and stability even under abrupt motion changes. The probabilistic data association (PDA) framework makes the automatic selection of measurement sets possible, resulting in enhanced robustness to occlusions and moving objects. As the PDA associates stereo correspondences probabilistically, the explicit establishment of stereo matches is not necessary except during initialization, and the point features presence in the outer region of the stereo image pair can be utilized. The performance is demonstrated by applying the pose information to control cameras in a virtual environment.

In this work, we solve the pose estimation problem for robot motion by placing multiple cameras on the robot. In particular, we use four cameras arranged as two back-to-back stereo pairs combined with the extended Kalman filter (EKF). The reason for using multiple cameras is that the pose estimation problem is more constrained for multiple cameras than for a single camera. Back-to-back cameras are used since they provide more information. Stereo information is used in self initialization and outlier rejection. Different approaches to solve the long-sequence-drift have been suggested. Both the simulations and the real experiments show that our approach is fast, robust, and accurate.

Traditional vision-based 3-D motion estimation algorithms for robots require given or calculated 3-D models while the motion is being tracked. We propose a high-speed extended-Kalman-filter-based approach that recovers position and orientation from stereo image sequences without prior knowledge as well as the procedure for the reconstruction of 3-D structures. Empowered by the use of the trifocal tensor, the computation step of 3-D models can be eliminated. The algorithm is thus more flexible and can be applied to a wide range of domains. The twist motion model is also adopted to parameterize the 3-D motion such that the motion representation in the proposed algorithm is robust and minimal. As the number of parameters to be estimated is reduced, our algorithm is more efficient, stable and accurate compared to traditional approaches. The proposed method has been verified using a real image sequence with ground truth.

Resolution improvement from several images is typically restricted to simple planar rotations and translations. In this paper, a super-resolution algorithm that allows 3D ego-motion of the camera system is proposed. By exploiting the trifocal tensor constraint of a stereo camera system, the intermediate step of scene structure recovery is effectively skipped. 3D motion is estimated recursively through an extended Kalman filter and used by a novel image warping procedure to perform resolution enhancement. Real image experiment with comparison to well known Irani-Peleg approach confirms the validity of the algorithm.

An innovative extended Kalman filter (EKF) algorithm for pose tracking has been proposed in this paper. It has the advantages of both structure and motion-based (SAM-based) and traditional model-based pose estimation algorithms. With no prior information about the scene, the pose sequence can be computed directly from images while the updating of the 3-D structure is not necessary. To achieve the goal, a constant velocity motion model is used as the dynamic system and the trifocal tensor point transfer function is applied to the measurement model of the filter. The resulting algorithm is stable, accurate and efficient. An empirical comparison with existing EKFs which deal with the same problem has been made and shows that our approach outperformed the others. The proposed method has been tested with various video sequences to demonstrate its performance in real situations.

Inserting synthetic objects into video sequences has gained much interest in recent years. Fast and robust vision-based algorithms are necessary to make such an application possible. Traditional pose tracking schemes using recursive structure from motion techniques adopt one Kalman filter and thus only favor a certain type of camera motion. We propose a robust simultaneous pose tracking and structure recovery algorithm using the interacting multiple model (IMM) to improve performance. In particular, a set of three extended Kalman filters (EKFs), each describing a frequently occurring camera motion in real situations (general, pure translation, pure rotation), is applied within the IMM framework to track the pose of a scene. Another set of EKFs,one filter for each model point, is used to refine the positions of the model features in the 3-D space. The filters for pose tracking and structure refinement are executed in an interleaved manner. The results are used for inserting virtual objects into the original video footage. The performance of the algorithm is demonstrated with both synthetic and real data. Comparisons with different approaches have been performed and show that our method is more efficient and accurate.

In this paper, an innovative extended Kalman filter (EKF) algorithm for pose tracking using the trifocal tensor is proposed. In the EKF, a constant-velocity motion model is used as the dynamic system, and the trifocal-tensor constraint is incorporated into the measurement model. The proposed method has the advantages of those structure- and-motion-based approaches in that the pose sequence can be computed with no prior information on the scene structure. It also has the strengths of those model-based algorithms in which no updating of the three-dimensional (3-D) structure is necessary in the computation. This results in a stable, accurate, and efficient algorithm. Experimental results show that the proposed approach outperformed other existing EKFs that tackle the same problem. An extension to the pose-tracking algorithm has been made to demonstrate the application of the trifocal constraint to fast recursive 3-D structure recovery.

Finding the pose and structure of an unknown object from an image sequence has many applications in graphics, virtual reality, and multimedia processing. In this paper, we address this problem by using a two-stage iterative method. Starting from an initial guess of the structure, the first stage estimates the pose of the object. The second stage uses the estimated pose information to refine the structure. This process is repeated until the difference between the observed data and data re-projected from the estimated model is minimized. This method is a variation of the classical bundle adjustment method, but is faster in execution and is simpler to implement. We used the Kanade-Lucas-Tomasi feature tracker for obtaining the image features. Synthetic and real data have been tested with good results.

This paper describes a genetic algorithm that tackles the pose-estimation problem in computer vision. Our genetic algorithm can find the rotation and translation of an object accurately when the three-dimensional structure of the object is given. In our implementation, each chromosome encodes both the pose and the indexes to the selected point features of the object. Instead of only searching for the pose as in the existing work, our algorithm, at the same time, searches for a set containing the most reliable feature points in the process. This mismatch filtering strategy successfully makes the algorithm more robust under the presence of point mismatches and outliers in the images. Our algorithm has been tested with both synthetic and real data with good results. The accuracy of the recovered pose is compared to the existing algorithms. Our approach outperformed the Lowe's method and the other two genetic algorithms under the presence of point mismatches and outliers. In addition, it has been used to estimate the pose of a real object. It is shown that the proposed method is applicable to augmented reality applications.

A recursive two-step method to recover structure and motion from image sequences based on Kalman filtering is described in this paper. The algorithm consists of two major steps. The first step is an extended Kalman filter (EKF) for the estimation of the object's pose. The second step is a set of EKFs, one for each model point, for the refinement of the positions of the model features in the three-dimensional (3-D) space. These two steps alternate from frame to frame. The initial model converges to the final structure as the image sequence is scanned sequentially. The performance of the algorithm is demonstrated with both synthetic data and real-world objects. Analytical and empirical comparisons are made among our approach, the interleaved bundle adjustment method, and the Kalman filtering-based recursive algorithm by Azarbayejani and Pentland. Our approach outperformed the other two algorithms in terms of computation speed without loss in the quality of model reconstruction.

A robust simultaneous pose tracking and structure recovery algorithm based on the Interacting Multiple Model (IMM) for augmented reality applications is proposed in this paper. A set of three extended Kalman filters (EKFs), each describes a frequently occurring camera motion in real situations (general, pure translation, pure rotation), is applied within the IMM framework to track the pose of an object. Another set of EKFs, one filter for each model point, is used to refine the positions of the model features in the 3D space. The filters for pose tracking and structure refinement are executed in an interleaved manner. The results are used for inserting virtual objects into the original video footage. The performance of the algorithm is demonstrated with both synthetic and real data. Comparisons with different approaches have been performed and show that our method is more efficient and accurate.

In this paper, a novel algorithm for creating virtual indoor environments is described. First, a panoramic mosaic is generated from a series of photos taken with a camera rotates along a horizontal axis. Then, from the panoramic mosaic image, a nonfixed viewing point virtual walkthrough system can be created by defining manually the corners in the vertical panoramic mosaic. The side ratio of the virtual walkthrough system can be obtained from the panning angle subsequently. By applying the cylindrical projection technique, the texture for the sides of the virtual walkthrough environment can be projected in a more realistic way. Real images have been used to verify our proposed algorithm with satisfactory results.

Consider a color camera mounted at a fixed point and a target object to be tracked. We propose a method to replace the common "blue screen" technique in performing segmentation of the object. The target object image in the video sequence is separated from the background through a segmentation process employing the color mixture model techniques. The background information is recovered through a similarity search with the panorama of the original background . The foreground segment can then be encoded by traditional compression while the scene background is represented as a panorama. Finally, the foreground object combines with the corresponding panoramic segment or any desired image on-the-fly to reconstruct the video frame. Our system can be used in low bandwidth applications as well as special demonstration in which the demonstrator can appear/disappear at any time he/she wishes.

A number of algorithms have been developed to determine the motion of rigid objects. Most of these algorithms work on single or multiple objects with single motion. However, in the real world, multiple motions are common place. In this paper, a novel algorithm for recovering the 3D motions of multiple moving rigid objects is derived. This new approach is able to determine the total number of motions in the system and to perform 3D motion-based segmentation by using incremental clustering. Then the motion of each independent rigid object can be recovered.