This paper presents a novel method that acquires camera position and orientation from a stereo image sequence without prior knowledge of the scene. To make the algorithm robust, the interacting multiple model probabilistic data association filter (IMMPDAF) is introduced. The interacting multiple model (IMM) technique allows the existence of more than one dynamic system in the filtering process and in return leads to improved accuracy and stability even under abrupt motion changes. The probabilistic data association (PDA) framework makes the automatic selection of measurement sets possible, resulting in enhanced robustness to occlusions and moving objects. In addition to the IMMPDAF, the trifocal tensor is employed in the computation so that the step of reconstructing the 3-D models can be eliminated. This further guarantees the precision of estimation and computation efficiency. Real stereo image sequences have been used to test the proposed method in the experiment. The recovered 3-D motions are accurate in comparison with the ground truth data and have been applied to control cameras in a virtual environment.

The novelty of this paper is the introduction of the interacting multiple model probabilistic data association filter (IMMPDAF) to the pose tracking problem. The interacting multiple model (IMM) technique allows the existence of more than one dynamic system in the filtering process and in return leads to improved accuracy and stability even under abrupt motion changes. The probabilistic data association (PDA) framework makes the automatic selection of measurement sets possible, resulting in enhanced robustness to occlusions and moving objects. As the PDA associates stereo correspondences probabilistically, the explicit establishment of stereo matches is not necessary except during initialization, and the point features presence in the outer region of the stereo image pair can be utilized. The performance is demonstrated by applying the pose information to control cameras in a virtual environment.

Traditional vision-based 3-D motion estimation algorithms for robots require given or calculated 3-D models while the motion is being tracked. We propose a high-speed extended-Kalman-filter-based approach that recovers position and orientation from stereo image sequences without prior knowledge as well as the procedure for the reconstruction of 3-D structures. Empowered by the use of the trifocal tensor, the computation step of 3-D models can be eliminated. The algorithm is thus more flexible and can be applied to a wide range of domains. The twist motion model is also adopted to parameterize the 3-D motion such that the motion representation in the proposed algorithm is robust and minimal. As the number of parameters to be estimated is reduced, our algorithm is more efficient, stable and accurate compared to traditional approaches. The proposed method has been verified using a real image sequence with ground truth.

Resolution improvement from several images is typically restricted to simple planar rotations and translations. In this paper, a super-resolution algorithm that allows 3D ego-motion of the camera system is proposed. By exploiting the trifocal tensor constraint of a stereo camera system, the intermediate step of scene structure recovery is effectively skipped. 3D motion is estimated recursively through an extended Kalman filter and used by a novel image warping procedure to perform resolution enhancement. Real image experiment with comparison to well known Irani-Peleg approach confirms the validity of the algorithm.

An innovative extended Kalman filter (EKF) algorithm for pose tracking has been proposed in this paper. It has the advantages of both structure and motion-based (SAM-based) and traditional model-based pose estimation algorithms. With no prior information about the scene, the pose sequence can be computed directly from images while the updating of the 3-D structure is not necessary. To achieve the goal, a constant velocity motion model is used as the dynamic system and the trifocal tensor point transfer function is applied to the measurement model of the filter. The resulting algorithm is stable, accurate and efficient. An empirical comparison with existing EKFs which deal with the same problem has been made and shows that our approach outperformed the others. The proposed method has been tested with various video sequences to demonstrate its performance in real situations.

In this paper, an innovative extended Kalman filter (EKF) algorithm for pose tracking using the trifocal tensor is proposed. In the EKF, a constant-velocity motion model is used as the dynamic system, and the trifocal-tensor constraint is incorporated into the measurement model. The proposed method has the advantages of those structure- and-motion-based approaches in that the pose sequence can be computed with no prior information on the scene structure. It also has the strengths of those model-based algorithms in which no updating of the three-dimensional (3-D) structure is necessary in the computation. This results in a stable, accurate, and efficient algorithm. Experimental results show that the proposed approach outperformed other existing EKFs that tackle the same problem. An extension to the pose-tracking algorithm has been made to demonstrate the application of the trifocal constraint to fast recursive 3-D structure recovery.

In this paper, a novel algorithm for creating virtual indoor environments is described. First, a panoramic mosaic is generated from a series of photos taken with a camera rotates along a horizontal axis. Then, from the panoramic mosaic image, a nonfixed viewing point virtual walkthrough system can be created by defining manually the corners in the vertical panoramic mosaic. The side ratio of the virtual walkthrough system can be obtained from the panning angle subsequently. By applying the cylindrical projection technique, the texture for the sides of the virtual walkthrough environment can be projected in a more realistic way. Real images have been used to verify our proposed algorithm with satisfactory results.

Consider a color camera mounted at a fixed point and a target object to be tracked. We propose a method to replace the common "blue screen" technique in performing segmentation of the object. The target object image in the video sequence is separated from the background through a segmentation process employing the color mixture model techniques. The background information is recovered through a similarity search with the panorama of the original background . The foreground segment can then be encoded by traditional compression while the scene background is represented as a panorama. Finally, the foreground object combines with the corresponding panoramic segment or any desired image on-the-fly to reconstruct the video frame. Our system can be used in low bandwidth applications as well as special demonstration in which the demonstrator can appear/disappear at any time he/she wishes.

A new data representation of image-based objects is presented. With this representation, the user can change the illumination as well as the viewpoint of an image-based scene. Physically correct imagery can be generated without knowing any geometrical information (e.g. depth or surface normal) of the scene. By treating each pixel on the image plane as a surface element, we can measure its apparent BRDF (bidirectional reflectance distribution function) by collecting information in the sampled images. These BRDFs allow us to calculate the correct pixel colour under a new illumination set-up by fitting the intensity, direction and number of the light sources. We demonstrate that the proposed representation allows re-rendering of the scene illuminated by different types of light sources. Moreover, two compression schemes, spherical harmonics and discrete cosine transform, are proposed to compress the huge amount of tabular BRDF data. © 1998 John Wiley & Sons, Ltd.

We present a new scheme of data representation for image-based objects. It allows the illumination to be changed interactively without knowing any geometrical information (e.g. depth or surface normal) of the scene, but the resulting images are physically correct. The scene is first sampled from different view points and under different illuminations. By treating each pixel on the image plane as a surface element, the sampled images are used to measure the apparent BRDF of each surface element. Two compression schemes, spherical harmonics and discrete cosine transform, are proposed to compress the tabular BRDF data. Whenever the user changes the illumination a certain number of views are reconstructed. The correct user perspective view is then displayed using the standard texture mapping hardware. Hence, the intensity, the type and the number of the light sources can be manipulated interactively.