Mixed Reality (MR) can merge real and virtual worlds seamlessly. This paper proposes a method to realize smooth collaboration using a remote MR, which makes it possible for geographically distributed users to share the same objects and communicate in real time as if they are at the same place. In this paper, we consider a situation where the users at local and remote sites perform a collaborative work, and real objects to be operated exist only at the local site. It is necessary to share the real objects between the two sites. However, prior studies have shown sharing real objects by duplication is either too costly or unrealistic. Therefore, we propose a method to share the objects by virtualizing the real objects using Computer Vision (CV) and then rendering the virtualized objects using MR. We have proposed a remote collaborative work system to create a smoother user experience for collaborative work with virtualized objects for remote users. Through experiments, we confirmed the effectiveness of our approach.

This paper proposes an interactive display system that displays both of the exterior and interior construction of a building diorama by using a projection-based Mixed-Reality (MR) technique, which is useful for understanding the complex construction and the spatial relationships between outside and inside. The users can hold and move the diorama model using their hands/body motion, so that they can observe the model from their favorite viewpoint. Our system obtains both of the user's information (the viewpoint and the gesture) and the diorama model's information (the pose) in 3D space by using two RGB-D cameras. The CG image corresponding to the user's viewpoint, gesture and the pose of the diorama is rendered by Dual Rendering algorithm in real time. As the result, the generated CG image is projected onto the diorama to realize MR display. We confirm the effectiveness of our proposed method by developing a pilot system.

We propose a new computer vision approach to locate a walking pedestrian by a camera image of first-person vision in practical situation. We assume reference points have been registered with other first-person vision images. We utilize SURF and define seven matching criteria that derive from the property of first-person vision so that it rejects false matching. We have implemented a preliminary system that can respond to a query within 1/2 seconds for a path of approximately 1 km long around Tokyo downtown area where pedestrians and vehicles are always in images.

Our research aims to generate a player's view video stream by using a 3D free-viewpoint video technique. Since player trajectories are necessary to generate the video, we propose a real-time player trajectory estimation method by utilizing the shadow regions from soccer scenes. This paper describes our trial to realize real-time processing. We divide the process into capture and server computers. In addition, we reduced the processing cost with pipeline parallelization and optimization. We apply our proposed method to an actual soccer match held in a stadium and show its effectiveness.

This paper proposes a method to estimate the trajectories of soccer players by using two cameras set in a large-scale outdoor space such as a soccer stadium, which is normally not advantageous for image processing. We estimate a player¿s position as the intersection of the primary axes of two body regions, corresponding to the two cameras, and a shadow region on the surface of the field. By projecting the foreground silhouette regions extracted from both cameras¿ images onto the soccer field, each player¿s body region and its shadow region are identified. Furthermore, our method utilizes color information of the players' uniforms to improve the accuracy of object tracking. We have applied our proposed method to a real soccer game held in a soccer stadium and demonstrated its effectiveness.

When human cannot perceive the inconsistency of artificial shadows which are not physically correct, they are acceptable as “perceptually-correct” shadows. This paper focuses on the simplification of light-source models for generating the perceptually-correct artificial shadows. First, we conducted subjective evaluations to obtain knowledge about the human perception of the shadows. Then the knowledge was applied to control the resolution of the light-source map to generate perceptually-correct artificial shadows. Comparative studies among artificial and real shadows justified perceptually correctness. All experiments were done using still images, not videos. Our research becomes a reference to determine the resolution of light-source map in an MR scene.

We propose a new on-line modeling and rendering method for visualizing players in sports. This method targets the intricate geometry of players in sports such as wrestling. Our system can maintain modeling and rendering quality on producing free-viewpoint 3D video of the players by utilizing the virtual viewpoint information of the viewer. Viewers can move a virtual camera freely over the scene while the system manages the processing cost by changing the size of the voxels and microfacets used for spatial modeling of the players. The system first estimates the rough 3D shape of the players in voxel format, and then assigns a microfacet to each voxel. The texture of the microfacet is obtained from the video cameras surrounding the players. Our preliminary system was evaluated on the wrestling data taken in the Yoyogi National Gymnasium, Tokyo, Japan. We also conducted subjective evaluation of the free-viewpoint video thus produced and found a good balance of rendering quality and high frame-rate performance.

This paper proposes a novel remote face-to-face mixed reality (MR) system that enables two people in distant places to share MR space. Challenging issues to realize such an MR system include capturing, sending, and rendering each user's appearance in real time. We developed a method to represent user's upper body and hands on the table as a single deformed-billboard. An MR Othello game is implemented as a test bed of the remote face-to-face MR system. Users can play the tabletop game as if their opponent were sitting across from the table, despite being physically separated. By detecting and sending the status of each real game board to the other site, both users feel that they are sharing tabletop objects.

A Nested Marker, a novel visual marker for camera calibration in augmented reality (AR), enables accurate calibration even when the observer is moving very close to or far away from the marker. Our proposed Nested Marker has a recursive layered structure. One marker at an upper layer contains four smaller markers at the lower layer. Smaller markers can also have lower-layer markers nesting inside them. Each marker can be identified by its inside pattern, so the system can select a proper calibration parameter set for the marker. When the observer views the marker close-up, the lowest layer marker will work. When the observer views the marker from a distance, the top-layer marker will work. It is also possible to simultaneously utilize all visible markers in different layers for more stable calibration. Note that Nested Marker can be used in a standard ARToolkit framework. We have also developed an AR system to demonstrate the ability of Nested Marker

A mixed-reality face (MR face) is a mosaic face with real and virtual facial parts, presented by overlaying a virtual facial part on a real face using mixed-reality techniques. An MR face is an effective means to improve communication in mixed-reality space by restoring the eye expressions lost when wearing HMDs. Photometric registration between the real and virtual parts is important because our eyes are very sensitive, even to small changes in human faces. However, efforts to achieve perfect 'physical' photometric registration on an MR face are not feasible in an ordinary MR space. Therefore, it is essential to clarify the sensitivity of our eyes to the photometric inconsistencies on an MR face, and to concentrate on resolving them. In this paper, we first present the results of a systematic experiment that evaluated our sensitivity to the photometric inconsistencies on an MR face. Then, a technique to resolve the inconsistency and an experimental system to demonstrate the effectiveness of an MR face are described.

If a pattern is projected onto a concave screen,the desired view cannot be correctly observed due to the influence of inter-reflections. This paper proposes a simple but effective technique for photometric compensation in consideration of inter-reflections. The compensation is accomplished by canceling inter-reflections estimated by the radiosity method. The significant advantage of our method is that iterative calculations are not necessary because it analytically solves the inverse problem of inter-reflections.

The concept of "enhanced eyes" to restore gaze awareness in a collaborative mixed-reality space is proposed. Three "enhanced eyes" schemes are described: controlling the highlight in the eyes to aid awareness of eye contact, deforming the eyelids to enhance eye motion, and adjusting the rotation angle of the eyeballs to improve perception of gaze direction. The effectiveness of the schemes has been confirmed by subjective evaluations. The "enhanced eyes" emulate the natural appearance of the face and are effective not only to indicate gaze direction but also to create the feeling of gaze.

We present a new video browsing method for multiple videos that are taken at large-scale space for live 3D events such as soccer games. By our method, multiple viewers over a computer network can browse a live 3D event from any viewpoint and each viewer can move his/her viewpoint freely. Our algorithm consists of five steps. Our system first captures videos from multiple cameras, then extracts texture segments from the videos, selects appropriate segments according to a viewpoint which is given by user dynamically, transmits them to users, and lays out the segments in virtual space so that each viewer can see the segments in a virtual environment as if the viewers were in the event. Our 3D video display system requires 10 Mbps at most to browse a soccer game. We conducted experiments at two real soccer stadiums and succeeded in realizing live realistic visualization with free viewpoint at about 26 fps.

In desktop manipulations such as cooking or assembly instruction videos, a performer often draws the viewers' attention to significant portions of the video's content by means of a specific set of physical behaviors. We verified the efficacy of videos edited based on those behaviors by observing television programs and simulating them with the use of our editing scheme. We first discuss the typical behaviors commonly shown in TV programs, and statistically describe the relationships between certain types of shot changes. We then propose an editing method using those behaviors, and demonstrate that our method generates videos whose quality is close to that of some of television programs. Moreover, we demonstrate the usability of our online editing system in experiments involving actual scenes.

This paper presents a new outdoor mixed-reality system designed for people who carry a camera-attached small handy device in an outdoor scene where a number of surveillance cameras are embedded. We propose a new functionality in outdoor mixed reality that the handy device can display live status of invisible areas hidden by some structures such as buildings, walls, etc. The function is implemented on a camera-attached, small handy subnotebook PC (HPC). The videos of the invisible areas are taken by surveillance cameras and they are precisely overlapped on the video of HPC camera, hence a user can notice objects in the invisible areas and see directly what the objects do. We utilize surveillance cameras for two purposes. (1) They obtain videos of invisible areas. The videos are trimmed and warped so as to impose them into the video of the HPC camera. (2) They are also used for updating textures of calibration markers in order to handle possible texture changes in real outdoor world. We have implemented a preliminary system with four surveillance cameras and proved that our system can visualize invisible areas in real time.

This paper proposes a method to realize a 3D video display system that can capture video from multiple cameras, reconstruct 3D models and transmit 3D video data in real time. We represent a target object with a simplified 3D model consisting of a single plane and a 2D texture extracted from multiple cameras. This 3D model is simple enough to be transmitted via a network. We have developed a prototype system that can capture multiple videos, reconstruct 3D models, transmit the models via a network, and display 3D video in real time. A 3D video of a typical soccer scene that includes a dozen players was processed at 26 frames per second.

A live 3D video system in a real soccer stadium is presented. Players on the pitch are represented with a simplified 3D model. The model is reconstructed from multiple videos obtained by nine CCD cameras surrounding the pitch. Observers can watch the game from arbitrary viewing position. By using real textures of the players, realistic video presentation is possible. All processes are fully automatic and real time. The system can transmit live 3D video to distant places.

The authors introduce their research for realizing a 3D video display system in a very large-scale space such as a soccer stadium, concert hall, etc. They propose a method for describing the shape of a 3D object with a set of planes in order to synthesize a novel view of the object effectively. The most effective layout of the planes can be determined based on the relative locations of an observer's viewing position, multiple cameras, and 3D objects. A method is described for controlling the LOD of the 3D representation by adjusting the orientation, interval, and resolution of planes. The data size of the 3D model and the processing time can be reduced drastically. The effectiveness of the proposed method is demonstrated by experimental results.

We propose a new scheme to recover the eye-contact between multiple users in a shared mixed-reality space. The eye-contact in a shared mixed-reality space is lost as the side effect of wearing head-mounted displays. We synthesize facial images in real-time with arbitrary poses and eye expressions by using several photographs of the user. The face images are overlaid in order to diminish the HMD in his partner's view for the recovery of eye-contact. The basic idea, facial image synthesis, and an experimental system to diminish HMD are presented in this paper.

We often need too much time for reading documents, since it is often difficult to efficiently grasp its outline. For this purpose, we propose our diagram generation scheme for presenting the structures of a text. The semantic structure of a tagged text is effectively translated to diagrams, and they are linked to the text. In this paper, first, we describe how semantic structures can be expressed by a diagram. Then, we propose our framework for automatic diagram generation from tagged texts to diagrams.

In this paper, we introduce an intelligent system for video recording. First, we categorized targets and purposes of shooting, and discuss the cameraworks appropriate for them. Then, we propose camera control algorithms to realize such cameraworks. Based on this idea, we built a prototype pan-tilt camera control system, in which multiple cameras with different purposes automatically track and shoot the targets. We evaluated our system through recording of some presentations on desktop manipulation. The effectiveness of our algorithm was verified through some experiments.

Introduces a method for analyzing video records which contain personal activities captured by a head mounted camera. This aims to support the user in retrieving the most important or relevant portions from the videos. For this purpose, we use the user's behaviors which appear when he/she pays attention to something. We define two types of those behaviors, one of which is "gaze at something in a short period" and the other is "staying and continuously see something". These behaviors and the focused object can be detected by estimating camera and object motion. We describe the details of the method and experiments in which the method was applied to ordinary events.

This paper proposes a polynocular stereo algorithm that is the integration of two algorithms with/without occlusion handling mechanism. The algorithm is useful to improve the sharpness of depth map at occluding boundaries obtained by video-rate stereo machines without occlusion handling capability. In order to realize the integration, we have developed an algorithm for detecting occlusion area and a correspondence algorithm with occlusion handling mechanism. Using the evaluation values that are computed in the correspondence search process without considering occlusion, occlusion area can be extracted with small additional computational cost. We have developed a polynocular stereo algorithm based on sort-oriented occlusion handling mechanism. Experimental results using ground-truthed stereo images show the effectiveness of the integration.

We propose a linear algorithm that is useful for realizing geometric registration between the view of a real scene and that of a virtual object in an image-based rendering framework. In a unified framework, the novel view synthesis of a virtual object based on three views' matching constraints and the recovery of the camera pose that is necessary for the base image selection can be performed. The feasibility of the algorithm is demonstrated by using ground-truth synthesized data and real scene data.

We introduce a unified framework for developing matching constraints of multiple affine views and rederive 2-view (affine epipolar geometry) and 3-view (affine image transfer) constraints within this framework. We then describe a new linear method for Euclidean motion and structure from 3 calibrated affine images, based on insight into the particular structure of these multiple-view constraints. Compared with the existing linear method of Huang and Lee (1989), the new method uses different and more appropriate constraints. It has no failure mode of the Euclidean factorisation method of Tomasi and Kanade (1992). We demonstrate the method on real image sequences.

We propose a method for classifying image edges caused by different physical phenomena, i.e. reflectance change, shadow, occlusion, etc., by using color information around the edge. We assumed several simple models for object spatial arrangements. For each of them, typical locus of RGB values along the normal direction of each edge segment is modeled. Each locus is parametrized by several features. In the classification of actual edges, the most plausible phenomenon is selected by checking the consistency between the parameters from an actual edge profile and those from each model. For the improvement of accuracy, Dempster-Shafer probability model is employed to deal with the above parameters that are often weak and uncertain. Experiments showed good performances.

In stereo algorithms with more than two cameras, the improvement of accuracy is often reported since they are robust against noise. However, another important aspect of the polynocular stereo, that is the ability of occlusion detection, has been paid less attention. We intensively analyzed the occlusion in the camera matrix stereo (SEA) and developed a simple but effective method to detect the presence of occlusion and to eliminate its effect in the correspondence search. By considering several statistics on the occlusion and the accuracy in the SEA, we derived a few base masks which represent occlusion patterns and are effective for the detection of occlusion. Several experiments using typical indoor scenes showed quite good performance to obtain dense and accurate depth maps even at the occluding boundaries of objects.

The individuality of a human face depends on the fine details of the facial components, and it is necessary to extract and to describe these detailed patterns in order to recognize human faces. We propose a method to describe the eye figure with small parameters by classifying their patterns to typical groups. First, an eye image is divided into parts such as eyelid and inner corner, and a set of 1-dimensional slit projections is obtained from the 2-dimensional intensity array. Then, the principal component analysis is applied to these projections to find the major axes which have typical features. The individuality of each eye is parameterized by the principal component scores. The effectiveness of the description is evaluated by generating sketch images based on the parameters extracted from real eye images.

We propose a method to analyze detailed patterns on contour shapes. In this method, we express detailed patterns using wavelet local extrema which are obtained through wavelet transforms. Based on this description, two features of detailed patterns, the properties of small fractions (or corners) constituting the detailed patterns and the arrangement of these corners are extracted. Using these two important features, the similarities between two detailed patterns can be examined. The proposed method is applied to detailed patterns of leaf contours and some hand-drawn contours to show its feasibility.

In stereo algorithms with more than two cameras, the improvement of accuracy in correspondence search is often reported. On the other hand, another important aspect of the polynocular stereo is the ability of occlusion detection. The camera matrix stereo SEA, which we have developed, offers a simple but effective framework to detect the presence of occlusion and to obtain reliable correspondence. SEA can produce a dense and accurate depth map with sharp object profiles. In this paper, we made some systematic comparison of several algorithms for occlusion detection in SEA. The results are quite interesting and reasonable. They are useful to design an actual polynocular stereo system.

A novel computational algorithm is proposed for color constancy suitable to robot vision. A robot, or a computer, can exactly memorize image information observed in the past. Then it is natural to use more than one image to achieve color constancy. In the algorithm, it is possible to recover the illumination color and the reflectance color only based on the RGB values of two objects identified on two images. It requires no specific assumption on the scene. Experiments show the validity of the proposed algorithm.>

A novel scheme is proposed to integrate multiple stereo algorithms in a cooperative framework. Each algorithm is implemented in a separate module and executed in parallel. The stereo correspondence obtained in each algorithm is stored in the module. Confidence of each stereo correspondence is evaluated based on the ambiguity in the search process and it is attached to the correspondence result. During the execution, each module communicates with other modules and offers subsets of its results to another by its request. The cooperation among the modules makes it possible not only to improve the performance of each algorithm but also to make the integrated system highly adaptive to various scenes. A system integrating three stereo matching algorithms which use different kinds of image features has been developed on a parallel computer to demonstrate the feasibility of the scheme.>

An almost occlusion-free trinocular stereo algorithm is described. It can cope with the occlusion problem when trying to obtain depth data of good accuracy by using a long stereo baseline. A third camera located midway between a pair of stereo cameras is used for the purpose. A correspondence search algorithmbased on two-level dynamic programming was developed to obtain an optimal correspondence among the three images. Experiments showed the validity of the algorithm.>

This paper presents a stereo matching algorithm using the dynamic programming technique. The stereo matching problem, that is, obtaining a correspondence between right and left images, can be cast as a search problem. When a pair of stereo images is rectified, pairs of corresponding points can be searched for within the same scanlines. We call this search intra-scanline search. This intra-scanline search can be treated as the problem of finding a matching path on a two-dimensional (2D) search plane whose axes are the right and left scanlines. Vertically connected edges in the images provide consistency constraints across the 2D search planes. Inter-scanline search in a three-dimensional (3D) search space, which is a stack of the 2D search planes, is needed to utilize this constraint. Our stereo matching algorithm uses edge-delimited intervals as elements to be matched, and employs the above mentioned two searches: one is inter-scanline search for possible correspondences of connected edges in right and left images and the other is intra-scanline search for correspondences of edge-delimited intervals on each scanline pair. Dynamic programming is used for both searches which proceed simultaneously: the former supplies the consistency constraint to the latter while the latter supplies the matching score to the former. An interval-based similarity metric is used to compute the score. The algorithm has been tested with different types of images including urban aerial images, synthesized images, and block scenes, and its computational requirement has been discussed.