Abstract In Mixed Reality (MR) applications, immersion of virtual objects in captured video contributes to the perceived unification of two worlds, one real, one synthetic. Since virtual actors and surround may appear both closer and farther than real objects, compositing must consider spatial relationships in the resulting world. Chroma keying, often called blue screening or green screening, is one common solution to this problem. This method is under‐constrained and most commonly addressed through a combination of environment preparation and commercial products. In interactive MR domains that impose restrictions on the video camera hardware, such as in experiences using video see‐through (VST) head‐mounted displays (HMD), chroma keying becomes even more difficult due to the relatively low camera quality, the use of m ltiple camera sources (one per eye), and the required processing speed. Dealing with these constraints requires a fast and affordable solution. In our approach, we precondition the chroma key by using principal component analysis (PCA) to obtain usable alpha mattes from video streams in real‐time on commodity graphics processing units (GPUs). In addition, we demonstrate how our method compares to off‐line commercial keying tools and how it performs with respect to signal noise within the video stream. Copyright © 2009 John Wiley & Sons, Ltd.

Recently, the concept of behavioral media using a mobile robot has emerged in tele-robotics, where the mobile robot behaves as a user's second body or an agent in a remote site. An important issue in behavioral media is how to control the robot intuitively immersing a user into a remote site. This paper describes a new approach which combines locomotion interface and immersive projection display of a remote site for controlling a networked mobile robot. We have developed a prototype system and have evaluated the usability of the system with experiments

Recently, generation of novel views from images acquired by multiple cameras has been investigated. It can be applied to telepresence effectively. Most conventional methods need some assumptions about the scene such as a static scene and limited positions of objects. In this paper, we propose a new method for generating novel view images of a dynamic scene with a wide view, which does not depend on the scene. The images acquired from omni-directional cameras are first divided into static regions and dynamic regions. The novel view images are then generated by applying a morphing technique to static regions and by computing visual hulls for dynamic regions in real-time. In experiments, we show that a prototype system can generate novel view images in real-time from live video streams.

With the spread of digital cameras, shooting photos has been becoming an everyday a ff air. However, there are few methods or systems to manage photos simply, and a huge amount of photo data remains unorganized. Although it is possible to add appropriate words explaining the contents of the photo as one of the methods to manage photos, it requires much time and e ff ort to input such indexes manually. It is also difficult to add indexes intended by a user automatically. This paper proposes a semi-automatic photo indexing system that enables users to generate indexes simply and browse a photo library efficiently. Index candidates are acquired from map database retrieval and relevant word extraction using web retrieval based on shooting position and orientation information. We have implemented an indexing system prototype based on the proposed method, and have carried out some experiments.

Recently, generation of novel views from images acquired by multiple cameras has been investigated in the fields of virtual and mixed reality. Most conventional methods need some assumptions about the scene such as a static scene and limited positions of objects. We propose a new method for generating novel view images of a dynamic scene with a wide view, which does not depend on the scene. The images acquired from omni-directional cameras are first divided into static regions and dynamic regions. The novel view images are then generated by applying a morphing technique to static regions and by computing visual hulls for dynamic regions in realtime. In experiments, we show that a prototype system can generate novel view images in real-time from live video streams.

This paper describes a new method for detecting moving objects with an omnidirectional camera. The proposed method uses a fixed omnidirectional camera which can acquire images surrounding the camera and a background subtraction technique based on adaptively updating background images. In subtracting the background image from the input image, we consider the oscillation of intensity so that the proposed method can robustly detect moving objects under the condition that there are fluorescent lights and incoming lights from windows. In experiments, moving objects have been successfully detected in indoor environments.

The progress in computer graphics has made it possible to construct various virtual environments such as urban or natural scenes. The paper proposes a hybrid method to construct a realistic virtual environment containing an existing real scene. The proposed method combines two different types of 3-D models. A 3-D geometric model is used to represent virtual objects in the user's vicinity, enabling a user to handle virtual objects. A texture mapped cylindrical 2.5-D model of a real scene is used to render the background of the environment, maintaining real-time rendering and increasing realistic sensation. The cylindrical 2.5-D model is generated from cylindrical stereo images captured by an omnidirectional stereo imaging sensor. A prototype mixed reality system has been developed to confirm the feasibility of the method, in which panoramic binocular stereo images are projected on a cylindrical immersive projective display depending on the user's view point in real time.

This paper presents a new method to generate and present arbitrarily directional binocular stereo images from a sequence of omnidirectional images. A sequence of omnidirectional images is taken by moving an omnidirectional image sensor in a static real environment. The motion of the omnidirectional image sensor is constrained to a plane. The sensor's route and speed are known. In the proposed method, a fixed length of the sequence is buffered in a computer to generate arbitrarily directional binocular stereo images by combining captured rays. Using the method a user can look around a scene in the distance with rich 3D sensation without significant time delay. This paper describes the principle of real-time generation of binocular stereo images. In addition, we introduce a prototype telepresence system of view-dependent stereo image generation and presentation.

We (1995) have proposed an omnidirectional image sensor called HyperOmni Vision. This sensor can acquire omnidirectional and perspective images in real time. In this paper we present a new Hough transform method for an omnidirectional image by using a cubic Hough space. Furthermore, we describe a method for reconstructing 3D line segments from an omnidirectional input image sequence obtained by HyperOmni Vision with known motion.

Described here is a visual navigation method for navigating a mobile robot along a man-made route such as a corridor or a street. We have proposed an image sensor, named HyperOmniVision, with a hyperboloidal mirror for vision based navigation of the mobile robot. This sensing system can acquire an omnidirectional view around the robot in real time. In the case of the man-made route, road boundaries between the ground plane and wall appears as a close-looped curve in the image. By making use of this optical characteristic the robot can avoid obstacles and move along the corridor by tracking the close-looped curve with an active contour model. Experiments that have been done in a real environment are described.

This study proposes a new factorization method for shape and motion recovery. In the past, the fourth greatest singular value of the measurement matrix was ignored. But when noise is large enough so that the fourth greatest singular value can not be ignored, it would be difficult to get reliable results by using the traditional factorization method. In order to acquire reliable results, We start with adopting an orthogonalization method to find a matrix which is composed of three mutually orthogonal vectors. By using this matrix, another matrix can be obtained. Then, the two expected matrices which represent shape of object and motion of camera/object, can be obtained through normalization. This study also conducts several experiments to discuss the feasibility of the proposed method.

We have developed a high-resolution omnidirectional stereo imaging sensor that can take images at video-rate. The sensor system takes an omnidirectional view by a component constructed of six cameras and a hexagonal pyramidal mirror and acquires stereo views by symmetrically connecting two sensor components. The paper describes a method of generating stereo panoramic images by using our sensor. First, the sensor system is calibrated; that is, twelve cameras are correctly aligned with pyramidal mirrors and the Tsai's method restores the radial distortion of each camera image. Stereo panoramic images are then computed by registering the camera images captured at the same time.

This paper describes a visual surveillance and monitoring system which is based on omnidirectional imaging and view-dependent image generation from omnidirectional video streams. While conventional visual surveillance and monitoring systems usually consist of either a number of fixed regular cameras or a mechanically controlled camera, the proposed system has a single omnidirectional video camera using a hyperboloidal mirror. This approach has an advantage of less latency in looking around a large field of view. In a prototype system developed, the viewing direction is determined by viewers' head tracking, by using a mouse, or by moving object trading in the omnidirectional image.

Described here is a method for estimating a rolling motion of a mobile robot from optical flows. We have proposed an image sensor with a hyperboloidal mirror for vision based navigation of the mobile robot. Its name is HyperOmniVision. This sensing system can acquire an omnidirectional view around the robot, in real-time, with use of the hyperboloidal mirror. The radial component of optical flow in HyperOmniVision has a periodic characteristic. The proposed method makes use of this characteristic to estimate robustly the rolling motion of the robot.

Described here is a method for estimating rolling and swaying motions of a mobile robot from optical flows. We have proposed an image sensor with a hyperboloidal mirror for vision based navigation of the mobile robot. Its name is HyperOmni Vision. This sensing system can acquire an omnidirectional view around the robot, in real-time, with use of the hyperboloidal mirror. The radial component of optical flow in HyperOmniVision has a periodic characteristic. The circumferential component of optical flow has a symmetric characteristic. The proposed method makes use of these characteristics to estimate robustly the rolling and swaying motions of the robot.

Described here is an image sensor with a hyperboloidal mirror for vision based navigation of a mobile robot. Its name is HyperOmni Vision. This sensing system can acquire an omnidirectional view around the robot, in real-time, with use of a hyperboloidal mirror. The authors show a prototype of a mobile robot system with HyperOmni Vision and a method for estimating the motion of the robot and finding unknown obstacles.

Described here is an image sensor with a hyperboloidal mirror for vision based navigation of a mobile robot. Its name is HyperOmni Vision. This sensing system can acquire an omnidirectional view around the robot, in real-time, with use of a hyperboloidal mirror.