Automatically generating a summary of a sports video poses the challenge of detecting interesting moments, or highlights, of a game. Traditional sports video summarization methods leverage editing conventions of broadcast sports video that facilitate the extraction of high-level semantics. However, user-generated videos are not edited and, thus, traditional methods are not suitable to generate a summary. In order to solve this problem, this paper proposes a novel video summarization method that uses players' actions as a cue to determine the highlights of the original video. A deep neural-network-based approach is used to extract two types of action-related features and to classify video segments into interesting or uninteresting parts. The proposed method can be applied to any sports in which games consist of a succession of actions. Especially, this paper considers the case of Kendo (Japanese fencing) as an example of a sport to evaluate the proposed method. The method is trained using Kendo videos with ground truth labels that indicate the video highlights. The labels are provided by annotators possessing a different experience with respect to Kendo to demonstrate how the proposed method adapts to different needs. The performance of the proposed method is compared with several combinations of different features, and the results show that it outperforms previous summarization methods.

The proliferation of off-the-shelf head-mounted displays (HMDs) let end-users enjoy virtual reality applications, some of which render a real-world scene using a novel view synthesis (NVS) technique. View-dependent texture mapping (VDTM) has been studied for NVS due to its photo-realistic quality. The VDTM technique renders a novel view by adaptively selecting textures from the most appropriate images. However, this process is computationally expensive because VDTM scans every captured image. For stereoscopic HMDs, the situation is much worse because we need to render novel views once for each eye, almost doubling the cost. This paper proposes light-weight VDTM tailored for an HMD. In order to reduce the computational cost in VDTM, our method leverages the overlapping fields of view between a stereoscopic pair of HMD images and pruning the images to be scanned. We show that the proposed method drastically accelerates the VDTM process without spoiling the image quality through a user study.

Augmented reality (AR) marker hiding is a technique to visually remove AR markers in a real-time video stream. A conventional approach transforms a background image with a homography matrix calculated on the basis of a camera pose and overlays the transformed image on an AR marker region in a real-time frame, assuming that the AR marker is on a planar surface. However, this approach may cause discontinuities in textures around the boundary between the marker and its surrounding area when the planar surface assumption is not satisfied. This paper proposes a method for AR marker hiding without discontinuities around texture boundaries even under nonplanar background geometry without measuring it. For doing this, our method estimates the dense motion in the marker's background by analyzing the motion of sparse feature points around it, together with a smooth motion assumption, and deforms the background image according to it. Our experiments demonstrate the effectiveness of the proposed method in various environments with different background geometries and textures.

Automatic sports video summarization poses the challenge of acquiring semantics of the original video, and existing work leverages various knowledge in application domains, e.g., structure of games and editing conventions. In this paper, we propose a personal sports video summarization method for self-recorded RGB-D videos, which became available to the public due to the commodification of off-the-shelf RGB-D sensors. We focus on sports whose games consist of a succession of actions and, unlike previous research, we use human action recognition on the depth sequences in order to acquire higher level semantics of the video. The recognition results are used along with an entropy-based activity measure to train a hidden Markov model of the highlights of different games to extract a summary from the original RGB-D video. We trained our novel highlights model with the subjective opinion of users with different experience in the sport. We took Kendo, a martial art, as an example sport to evaluate our method, and objectively/subjectively investigated the accuracy and quality of the generated summaries.

3D models of outdoor environments have been used for several applications such as a virtual earth system and a vision-based vehicle safety system. 3D data for constructing such 3D models are often measured by an on-vehicle system equipped with laser rangefinders, cameras, and GPS/IMU. However, 3D data of moving objects on streets lead to inaccurate 3D models when modeling outdoor environments. To solve this problem, this paper proposes a moving object detection method for point clouds by minimizing an energy function based on photometric and depth consistencies assuming that input data consist of synchronized point clouds, images, and camera poses from a single sequence captured with a moving on-vehicle system.

Diminished reality aims to remove real objects from video images and fill in the missing regions with plausible background textures in real time. Most conventional methods based on image inpainting achieve diminished reality by assuming that the background around a target object is almost planar. This paper proposes a new diminished reality method that considers background geometries with less constraints than the conventional ones. In this study, we approximate the background geometry by combining local planes, and improve the quality of image inpainting by correcting the perspective distortion of texture and limiting the search area for finding similar textures as exemplars. The temporal coherence of texture is preserved using the geometries and camera pose estimated by visual-simultaneous localization and mapping (SLAM). The mask region that includes a target object is robustly set in each frame by projecting a 3D region, rather than tracking the object in 2D image space. The effectiveness of the proposed method is successfully demonstrated using several experimental environments.

This paper proposes an example-based method to increase the resolution of a low-resolution image. In the proposed method, we generate example images by a novel view synthesis technique using 3D geometry reconstruction and camera pose estimation from a video or images capturing the same scene. We then increase the resolution by minimizing an energy function by searching for the optimal example from the generated example images. The proposed method has less limitations on camera positions and geometry of the target scene than those in conventional methods. Experiments demonstrate the effectiveness of the proposed method by qualitatively comparing the results of the proposed and conventional methods.

An enormous number of images are currently shared through social networking services such as Facebook. These images usually contain appearance of people and may violate the people's privacy if they are published without permission from each person. To remedy this privacy concern, visual privacy protection, such as blurring, is applied to facial regions of people without permission. However, in addition to image quality degradation, this may spoil the context of the image: If some people are filtered while the others are not, missing facial expression makes comprehension of the image difficult. This paper proposes an image melding-based method that modifies facial regions in a visually unintrusive way with preserving facial expression. Our experimental results demonstrated that the proposed method can retain facial expression while protecting privacy.

Recent popularization of camera devices, including action cams and smartphones, enables us to record videos in everyday life and share them through the Internet. Video blog is a recent approach for sharing videos, in which users enjoy expressing themselves in blog posts with attractive videos. Generating such videos, however, requires users to review vast amount of raw videos and edit them appropriately, which keeps users away from doing so. In this paper, we propose a novel video summarization method for helping users to create a video blog post. Unlike typical video summarization methods, the proposed method utilizes the text, which is written for a video blog post, and makes the video summary consistent with the content of the text. For this, we perform video summarization by solving an optimization problem, in which an objective function involves the content similarity between the summarized video and the text. Our user study with 20 participants has demonstrated that our proposed method is suitable to create video blog posts compared with conventional methods for video summarization.

When observing a person (an actor) performing or demonstrating some activity for the purpose of learning the action, it is best for the viewers to be present at the same time and place as the actor. Otherwise, a video must be recorded. However, conventional video only provides two-dimensional (2D) motion, which lacks the original third dimension of motion. In the presence of some ambiguity, it may be hard for the viewer to comprehend the action with only two dimensions, making it harder to learn the action. This paper proposes an augmented reality system to reenact such actions at any time the viewer wants, in order to aid comprehension of 3D motion. In the proposed system, a user first captures the actor's motion and appearance, using a single RGB-D camera. Upon a viewer's request, our system displays the motion from an arbitrary viewpoint using a rough 3D model of the subject, made up of cylinders, and selecting the most appropriate textures based on the viewpoint and the subject's pose. We evaluate the usefulness of the system and the quality of the displayed images by user study.

The purpose of this study is to increase the face detection accuracy in vehicle cabin. Although existing face detectors employed in consumer applications already have sufficient face detection accuracy for many situations, we revealed that detection rate of existing face detector is drastically decreased by shadow on the driver's face caused by sunlight whose relative direction to the driver is continuously changed while driving. In order to overcome this problem, we increase the number of driver's faces in training dataset by synthesizing the shadowed driver's faces from various directions of sunlight which are created using an image composing technique. In experiment, we found that the 20% to 40% shadowed faces should be blended into training dataset in terms of the generality and the adaptability for robust drivers' face detection.

Indirect augmented reality (IAR) utilizes pre-captured omnidirectional images and offline superimposition of virtual objects for achieving high-quality geometric and photometric registration. Meanwhile, IAR causes inconsistency between the real world and the pre-captured image. This paper describes the first-ever study focusing on the temporal inconsistency issue in IAR. We propose a novel IAR system which reflects real-world illumination change by selecting an appropriate image from a set of images pre-captured under various illumination. Results of a public experiment show that the proposed system can improve the realism in IAR.

Development of algorithms for vehicle safety systems, which support safety driving, takes a long period of time and a huge cost because it requires an evaluation stage where huge combinations of possible driving situations should be evaluated by using videos which are captured beforehand in real environments. In this paper, we address this problem by using free viewpoint images instead of the real images. More concretely, we generate free-viewpoint images from a combination of a 3D point cloud measured by laser scanners and an omni-directional image sequence acquired in a real environment. We basically rely on the 3D point cloud for geometrically correct virtual viewpoint images. In order to remove the holes caused by the unmeasured region of the 3D point cloud and to remove false silhouettes in surface reconstruction, we have developed a technique of free-viewpoint image generation that uses both a 3D point cloud and depth information extracted from images. In the experiments, we have evaluated our framework with a white line detection algorithm and experimental results have shown the applicability of free-viewpoint images for evaluation of algorithms.

A 3D modeling technique for an urban environment can be applied to several applications such as landscape simulations, navigational systems, and mixed reality systems. In this field, the target environment is first measured using several types of sensors (laser rangefinders, cameras, GPS sensors, and gyroscopes). A 3D model of the environment is then constructed based on the results of the 3D measurements. In this 3D modeling process, 3D points that exist on moving objects become obstacles or outliers to enable the construction of an accurate 3D model. To solve this problem, we propose a method for detecting 3D points on moving objects from 3D point cloud data based on photometric consistency and knowledge of the road environment. In our method, 3D points on moving objects are detected based on luminance variations obtained by projecting 3D points onto omnidirectional images. After detecting 3D the points based on evaluation value, the points are detected using prior information of the road environment.

Detecting important regions in videos has been extensively studied for past decades for their wide variety of applications including video summarization and retargeting. Visual attention models draw much attention for this purpose, which find visually salient regions. However, visual attention models ignore intentionally captured regions (ICRs) derived from videographers' intentions, i.e., what the videographers wanted to capture in their videos. This paper proposes a Markov random field-based ICR model for finding them. Observing that a videographer's intention is embedded into camera motion together with objects' motion, our ICR model uses point trajectory-based features to distinguish ICRs from non-ICRs. It also leverages spatial and temporal consistency of ICRs to improve the performance. We have experimentally demonstrated our ICR model's performance and the difference between ICRs and visually salient regions.

This paper proposes a teleoperation interface by which an operator can control a robot from freely configured viewpoints using realistic images of the physical world. The viewpoints generated by the proposed interface provide human operators with intuitive control using a head-mounted display and head tracker, and assist them to grasp the environment surrounding the robot. A state-of-the-art free-viewpoint image generation technique is employed to generate the scene presented to the operator. In addition, an augmented reality technique is used to superimpose a 3D model of the robot onto the generated scenes. Through evaluations under virtual and physical environments, we confirmed that the proposed interface improves the accuracy of teleoperation.

This paper proposes a new diminished reality method for 3D scenes considering background structures. Most conventional methods using image inpainting assumes that the background around a target object is almost planar. In this study, approximating the background structure by the combination of local planes, perspective distortion of texture is corrected and searching area is limited for improving the quality of image inpainting. The temporal coherence of texture is preserved using the estimated structures and camera pose estimated by Visual-SLAM.

Augmented reality (AR) images with virtualized real objects can be used for various applications. However, such AR image generation requires hand-crafted 3D models of that objects, which are usually not available. This paper proposes a view-dependent texture (VDT)- and view-dependent geometry (VDG)-based method for generating high quality AR images, which uses 3D models automatically reconstructed from multiple images. Since the quality of reconstructed 3D models is usually insufficient, the proposed method inflates the objects in the depth map as VDG to repair chipped object boundaries and assigns a color to each pixel based on VDT to reproduce the detail of the objects. Background pixel exposure due to inflation is suppressed by the use of the foreground region extracted from the input images. Our experimental results have demonstrated that the proposed method can successfully reduce above visual artifacts.

This study proposes a framework to photorealistically synthesize virtual objects and virtualized real-world. We combine the offline rendering of virtual objects and the free-viewpoint image generation to take advantage of the higher quality of offline rendering without the computational cost of online computer graphics (CG) rendering; i.e., it incurs only the cost of the online computation for the free-viewpoint image generation. In addition, the generation of structured viewpoints (e.g., at every grid point) reduces the computational costs required to online process.

This paper describes a method for acquiring full spherical high dynamic range (HDR) images with no missing areas by using two omni directional cameras mounted on the top and bottom of an unmanned airship. The full spherical HDR images are generated by combining multiple omni directional images that are captured with different shutter speeds. The images generated are intended for uses in telepresence, augmented telepresence, and image-based lighting.

ISMAR 2012: IEEE International Symposium on Mixed and Augmented Reality , Nov 5-8, 2012 , Atlanta, GA, USA

We have developed an augmented telepresence system which enables virtual tourism beyond time and space. Augmented telepresence provides a user with both the view of a remote location and related information using augmented reality techniques. This study deals with the geometric and photometric registration problems to generate movie-quality augmented omnidirectional videos automatically. The user can look around the scene from the sky above Heijo palace Site which is an ancient capital in Nara, Japan in the technical demonstration.

In this paper, we propose a novel surface completion method to generate plausible shapes and textures for missing regions of 3D models. The missing regions are filled in by minimizing two energy functions for shape and texture, which are both based on similarities between the missing region and the rest of the object; in doing so, we take into account the positive correlation between shape and texture. We demonstrate the effectiveness of the proposed method experimentally by applying it to two models.

This paper proposes a novel method for generating arbitrary stereoscopic view from multiple omni directional image sequences. Although conventional methods for arbitrary view generation with an image-based rendering approach can create binocular views, positions and directions of viewpoints for stereoscopic vision are limited to a small range. In this research, we attempt to generate arbitrary stereoscopic views from omni directional image sequences that are captured in various multiple paths. To generate a high-quality stereoscopic view from a number of images captured at various viewpoints, appropriate ray information needs to be selected. In this paper, appropriate ray information is selected from a number of omni directional images using a penalty function expressed as ray similarity. In experiments, we show the validity of this penalty function by generating stereoscopic view from multiple real image sequences.

This paper proposes a method for estimating extrinsic camera parameters using video images and position data acquired by GPS. In conventional methods, the accuracy of the estimated camera position largely depends on the accuracy of GPS positioning data because they assume that GPS position error is very small or normally distributed. However, the actual error of GPS positioning easily grows to the 10m level and the distribution of these errors is changed depending on satellite positions and conditions of the environment. In order to achieve more accurate camera positioning in outdoor environments, in this study, we have employed a simple assumption that true GPS position exists within a certain range from the observed GPS position and the size of the range depends on the GPS positioning accuracy. Concretely, the proposed method estimates camera parameters by minimizing an energy function that is defined by using the reprojection error and the penalty term for GPS positioning.

This study is concerned with a large-scale telepresence system based on remote control of mobile robot or aerial vehicle. The proposed system provides a user with not only view of remote site but also related information by AR technique. Such systems are referred to as augmented telepresence in this paper. Aerial imagery can capture a wider area at once than image capturing from the ground. However, it is difficult for a user to change position and direction of viewpoint freely because of the difficulty in remote control and limitation of hardware. To overcome these problems, the proposed system uses an autopilot airship to support changing user's viewpoint and employs an omni-directional camera for changing viewing direction easily. This paper describes hardware configuration for aerial imagery, an approach for overlaying virtual objects, and automatic control of the airship, as well as experimental results using a prototype system.

Surface completion is a technique for filling missing regions in 3D models measured by range scanners and videos. Conventionally, although missing regions were filled with the similar shape in a model, the completion process was fairly inefficient because the whole region in the model was searched for the similar shape. In this paper, the completion is efficiently performed using principal curvatures of local shape. In experiments, the effectiveness of the proposed method is successfully verified with subjective evaluation. In addition, the quantitative evaluation which has not been in the literature is newly performed.

In annotation overlay applications using augmented reality (AR), view management is widely used for improving readability and intelligibility of the annotations. In order to recognize the visible portions of objects in the user's view, the positions, orientations, and shapes of the objects should be known in the case of conventional view management methods. However, it is difficult for a wearable AR system to obtain the positions, orientations and shapes of objects because the target object is usually moving or non-rigid. In this paper, we propose a view management method to overlay annotations of moving or non-rigid objects for networked wearable AR. The proposed method obtains positions and shapes of target objects via a network in order to estimate the visible portions of the target objects in the user's view. Annotations are located by minimizing penalties related to the overlap of an annotation, occlusion of target objects, length of a line between the annotation and the target object, and distance of the annotation in sequential frames. Through experiments, we have proven that the prototype system can correctly provide each user with annotations on multiple users of wearable AR systems.

3D mesh models generated with range scanner or video images often have holes due to many occlusions by other objects and the object itself. This paper proposes a novel method to fill the missing parts in the incomplete models. The missing parts are filled by minimizing the energy function, which is defined based on similarity of local shape between the missing region and the rest of the object. The proposed method can generate complex and consistent shapes in the missing region. In the experiment, the effectiveness of the method is successfully demonstrated by applying it to complex shape objects with missing parts.

This paper describes a real-time pre-visualization method using augmented reality techniques for videographers. It enables them to test camera work without real actors in a real environment. As a substitute for real actors, virtual ones are superimposed on a live video in real-time according to a real camera motion and an illumination condition. The key technique of this method is real-time motion estimation of a camera, which can be applied to unknown complex environments including natural objects. In our method, geometric and photometric registration problems for such unknown environments are solved to realize the above visualization. A prototype system demonstrates availability of the pre-visualization method.

In the field of computer vision, many kinds of camera parameter estimation methods have been proposed. As one of these methods, an extrinsic camera parameter estimation method that uses pre-constructed feature landmark database has been studied. In this method, extrinsic camera parameters of video images are estimated from correspondences between landmarks and image features. Although this method can work in a large outdoor environment, its computational cost in matching process is expensive and it cannot work in real-time. In this paper, to achieve real-time camera parameter estimation, the number of matching candidates are reduced by using priorities of landmarks that are determined from previously captured video sequences.

This paper describes photometric registration for augmented reality (AR) using a high-dynamic-range (HDR) image. In photorealistic AR, estimating the lighting environment of virtual objects is difficult because of low dynamic range cameras. In order to overcome this problem, we propose a method that estimates the lighting environment from an HDR image and renders virtual objects using an HDR environment map. Virtual objects are overlaid in real-time by adjusting the dynamic range of the rendered image with tone mapping according to the exposure time of the camera. The HDR image is generated from multiple images captured with various exposure times. We have found through experimentation that the updating rate is improved by effectively limiting the dynamic range, depending on the exposure time. We have verified the effect of limiting the dynamic range on the reality of virtual objects.

We propose a user localization system that uses invisible markers for wearable augmented reality (AR) in large indoor environments. Wearable AR systems have received a great deal of attention as a new method for displaying location-based information in the real world. For using wearable AR systems, it is necessary to measure the position and orientation of a user using a positioning infrastructure without the undesirable visual effects that arise from merging real and virtual worlds. In addition, the infrastructure of the localization environment must be constructed easily and cheaply. The proposed system can estimate the position and orientation of a user precisely by affixing wallpapers containing printed invisible markers on ceilings or walls. The user's position and orientation are estimated by recognizing the markers using an infrared camera with infrared LEDs. To construct environments for the localization system, we developed an initialization tool that calibrates the alignment of the markers from photographs taken by flash illumination using a digital still camera.

This paper proposes an accurate geometric registration method by estimating blur effects from a degraded image with image markers for augmented reality. A small and inexpensive camera used in augmented reality systems sometimes captures degraded images because its focus and/or iris are fixed. This degradation of a captured image affects the accuracy of the detected positions of feature points in the image. The proposed method improves the accuracy of the estimated camera position and posture by estimating blur effects from the captured image, and by correcting the detected positions of feature points through the results. The effectiveness of the method is confirmed through experiments of corner estimation from simulated images and extrinsic camera parameter estimation from real images.

This paper investigates the problem of object surface reflectance modeling, which is sometimes referred to as inverse reflectometry, for photorealistic rendering and effective multimedia applications. A number of methods have been developed for estimating object surface reflectance properties in order to render real objects under arbitrary illumination conditions. However, it is still difficult to densely estimate surface reflectance properties in real-time. This paper describes a new method for real-time estimation of the non-uniform surface reflectance properties in the inverse rendering framework. Experiments are conducted in order to demonstrate the usefulness and the advantage of the proposed methods through comparative study

Recently, the concept of behavioral media using a mobile robot has emerged in tele-robotics, where the mobile robot behaves as a user's second body or an agent in a remote site. An important issue in behavioral media is how to control the robot intuitively immersing a user into a remote site. This paper describes a new approach which combines locomotion interface and immersive projection display of a remote site for controlling a networked mobile robot. We have developed a prototype system and have evaluated the usability of the system with experiments

Photometric registration is as important as geometric registration to generate a seamless augmented reality scene. Especially the difference in image quality between a real image and virtual objects caused by defocusing and motion blurring in capturing a real scene image easily exhibits the seam between real and virtual worlds. To avoid this problem in video see-through augmented reality, it is necessary to simulate the optical system of camera when virtual objects are rendered. This paper proposes an image composition method for video see-through augmented reality, which is based on defocusing and motion blurring estimation from the captured real image and rendering of virtual objects with blur effects. In experiments, the effectiveness of the proposed method is confirmed by comparing a real image with virtual objects rendered by the proposed method.

Recently, generation of novel views from images acquired by multiple cameras has been investigated. It can be applied to telepresence effectively. Most conventional methods need some assumptions about the scene such as a static scene and limited positions of objects. In this paper, we propose a new method for generating novel view images of a dynamic scene with a wide view, which does not depend on the scene. The images acquired from omni-directional cameras are first divided into static regions and dynamic regions. The novel view images are then generated by applying a morphing technique to static regions and by computing visual hulls for dynamic regions in real-time. In experiments, we show that a prototype system can generate novel view images in real-time from live video streams.

With the spread of digital cameras, shooting photos has been becoming an everyday a ff air. However, there are few methods or systems to manage photos simply, and a huge amount of photo data remains unorganized. Although it is possible to add appropriate words explaining the contents of the photo as one of the methods to manage photos, it requires much time and e ff ort to input such indexes manually. It is also difficult to add indexes intended by a user automatically. This paper proposes a semi-automatic photo indexing system that enables users to generate indexes simply and browse a photo library efficiently. Index candidates are acquired from map database retrieval and relevant word extraction using web retrieval based on shooting position and orientation information. We have implemented an indexing system prototype based on the proposed method, and have carried out some experiments.

Recently, generation of novel views from images acquired by multiple cameras has been investigated in the fields of virtual and mixed reality. Most conventional methods need some assumptions about the scene such as a static scene and limited positions of objects. We propose a new method for generating novel view images of a dynamic scene with a wide view, which does not depend on the scene. The images acquired from omni-directional cameras are first divided into static regions and dynamic regions. The novel view images are then generated by applying a morphing technique to static regions and by computing visual hulls for dynamic regions in realtime. In experiments, we show that a prototype system can generate novel view images in real-time from live video streams.

By realizing augmented reality on wearable computers, it is possible to overlay annotations on the real world based on the user's current position and orientation. However, it is difficult for the user to understand links between annotations and real objects intuitively when the scene is complicated or many annotations are overlaid at the same time. This work describes a view management method which emphasizes user-viewed real objects and their annotations using 3D models of the real scene. The proposed method highlights the objects viewed by the user. In addition, when the viewed object is occluded by other real objects, the object is complemented by using an image, which is made from 3D models, on the overlaid image.

This paper describes a method for modeling wide area outdoor environments by integrating omnidirectional range and color images. The proposed method effectively reconstructs the 3D models of outdoor environments by using omnidirectional laser rangefinder and omnidirectional multi-camera system (OMS). In this paper, we also give experimental results of 3D wide area reconstruction using the data acquired at 50 points in our campus.

In mixed reality, especially in augmented virtuality which virtualizes real objects, it is important to estimate object surface reflectance properties to render the objects under arbitrary illumination conditions. Though several methods have been explored to estimate the surface reflectance properties, it is still difficult to estimate surface reflectance parameters faithfully for complex objects which have nonuniform surface reflectance properties and exhibit interreflections. We describe a new method for densely estimating nonuniform surface reflectance properties of real objects constructed of convex and concave surfaces with interreflections. We use registered range and surface color texture images obtained by a laser rangefinder. Experiments show the usefulness of the proposed method.

This paper describes a new method for detecting moving objects with an omnidirectional camera. The proposed method uses a fixed omnidirectional camera which can acquire images surrounding the camera and a background subtraction technique based on adaptively updating background images. In subtracting the background image from the input image, we consider the oscillation of intensity so that the proposed method can robustly detect moving objects under the condition that there are fluorescent lights and incoming lights from windows. In experiments, moving objects have been successfully detected in indoor environments.

This paper describes a novel telepresence system that uses high-resolution movies and a reactive display system with a treadmill. In this system, users can walk through a virtualized environment by actually walking on a treadmill. According to walking motion which is detected by using 3-D position sensors put on both legs, the virtualized environment captured by an omnidirectional multi-camera system is projected on a multi-screen display.

This paper describes a wearable augmented reality system using positioning infrastructures and a pedometer. To realize augmented reality systems, the position and orientation of user's viewpoint should be obtained in real time. The proposed system measures the orientation of user's viewpoint by an inertial sensor and the user's position using positioning infrastructures in environments and a pedometer. The system specifies the user's position using the position ID received from RFID tags or IrDA markers which are the components of positioning infrastructures. When the user goes away from them, the user's position is alternatively estimated by using a pedometer. We have developed a navigation system using the proposed techniques and have proven the feasibility of the system with experiments.

In vision-based augmented reality systems, the relationship between the real and virtual worlds needs to be estimated to perform the registration of the virtual objects. This paper suggests a registration method which increases the registration depth of video see-through augmented reality systems using binocular cameras. The method uses both monocular and stereoscopic vision-based techniques in order to perform the registration. Also, the registration method can be combined with a facultative correction of the 2D positions of the feature points. The correction increases the stability and the accuracy of the proposed registration method.

In mixed reality, it is important to estimate object surface reflectance properties to render real objects under arbitrary illumination conditions. The authors developed a method to estimate reflectance properties of object surfaces densely. However, it was difficult to estimate surface reflectance properties faithfully for objects with interreflections. This paper describes a new method of densely estimating nonuniform surface reflectance properties of real objects constructed of convex and concave surfaces with interreflections. The proposed method first determines positions of light to take color images for discriminating diffuse and specular reflection components of surface reflection. Then, surface reflectance parameters are estimated based on radiosity. Experiments show the usefulness of the proposed method.

This paper proposes an augmented reality system with correct representation of shading and shadow. To realize a seamless augmented reality system, we need to resolve certain problems. The geometric and photometric registration problems are particularly important. These problems require the position of light sources and user's viewpoint. The proposed system resolves these problems using a 3D marker which combines a 2D square marker and a mirror ball. The 2D marker and the ball are used to estimate the relationship between the real and virtual worlds and the positions of light sources in the real world, respectively.

A macroblock-based optimization method is described for selecting quantization parameters so as to reduce the re-quantization errors in MPEG transcoding. Analysis has shown that re-quantization using a parameter larger than the original input quantization parameter but smaller than twice the original causes significant information loss in transcoding. This problem can be avoided by using a parameter larger than twice the original. The optimization method was thus extended based on a rate-distortion model to enable the re-quantization parameter to be selected from ones larger than twice the original input quantization range. This improved method outperformed a typical non-optimized transcoding scheme, improving the PSNR by about 1 dB.

Proposes a vision-based augmented reality system with a wide range of registration. To realize an augmented reality system, it is required to geometrically register real and virtual worlds. In the case of a vision-based augmented reality with marker tracking, its measurement range is usually limited because markers placed in the real world should be captured by cameras. The proposed method realizes a stereo vision-based augmented reality system with a wide range of registration by automatically detecting and tracking new, markers that come into sight. The feasibility of the system has been successfully demonstrated through experiments.

This paper describes a method of estimating the position and orientation of a camera for constructing a mixed reality (MR) system. In an MR system 3D virtual objects should be merged into a 3D real environment at a right position in real time. To acquire the user's viewing position and orientation is the main technical problem of constructing an MR system. The user's viewpoint can be determined by estimating the position and orientation of a camera using images taken at the viewpoint. Our method estimates the camera pose using screen coordinates of captured color fiducial markers whose 3D positions are known. The method consists of three algorithms for perspective n-points problems and rises each algorithm selectively. The method also estimates the screen coordinates of untracked markers that are occluded or are out of the view. It has been found that an experimental MR system that is based on the proposed method can seamlessly merge 3D virtual objects into a 3D real environment at right position in real-time and allows users to look around an area in which numbers are placed.

The progress in computer graphics has made it possible to construct various virtual environments such as urban or natural scenes. The paper proposes a hybrid method to construct a realistic virtual environment containing an existing real scene. The proposed method combines two different types of 3-D models. A 3-D geometric model is used to represent virtual objects in the user's vicinity, enabling a user to handle virtual objects. A texture mapped cylindrical 2.5-D model of a real scene is used to render the background of the environment, maintaining real-time rendering and increasing realistic sensation. The cylindrical 2.5-D model is generated from cylindrical stereo images captured by an omnidirectional stereo imaging sensor. A prototype mixed reality system has been developed to confirm the feasibility of the method, in which panoramic binocular stereo images are projected on a cylindrical immersive projective display depending on the user's view point in real time.

This paper presents a new method to generate and present arbitrarily directional binocular stereo images from a sequence of omnidirectional images. A sequence of omnidirectional images is taken by moving an omnidirectional image sensor in a static real environment. The motion of the omnidirectional image sensor is constrained to a plane. The sensor's route and speed are known. In the proposed method, a fixed length of the sequence is buffered in a computer to generate arbitrarily directional binocular stereo images by combining captured rays. Using the method a user can look around a scene in the distance with rich 3D sensation without significant time delay. This paper describes the principle of real-time generation of binocular stereo images. In addition, we introduce a prototype telepresence system of view-dependent stereo image generation and presentation.

We propose a gray-scale character recognition method for video indexing. It is robust against the problems of binarization against a complex background and low resolution. Unlike a traditional character recognition scheme through image binarization, we directly extract Gabor features (called Gabor jets) from video contents. The use of the Gabor filters contributes to freeing a tricky binarization process for cluttered images, and furthermore provides localized directional edge features, which have phase-shift invariance to edge positions. To form a feature vector to be classified, we accumulate the extracted Gabor features along projection lines in local regions, and then categorize them with a standard LVQ classifier. The projective accumulation provides robustness under character deformation caused by variation of font types or imprecise segmentation. We compare the proposed method by experiments with a typical OCR method, for which correct binarization is advantageously given. The proposed method shows similar or superior performance to the other method in understanding video captions.

To examine how computer technology can produce delicate, audience-inspiring presentations like "Noh", we attempted to create a Noh mask (Ko-omote) using a 3D computer graphics technique, by only referring to a carved wooden mask. It was possible to detect several characteristics of the mask's structure in the creation process. Rendering images showed the possibility that such a specific structure is closely related to effects of subtle facial expression changes. In order to bring more diverse effect to audiences, we also attempted to create a computer animation combining the mask model with story, movement and background images. This paper suggests the possibility of acquiring profound experiences with the perception and expression of "face" for both viewers and creators by using a computer graphics technique.

In an augmented reality system, it is required to obtain the position and orientation of the user's viewpoint in order to display the composed image while maintaining a correct registration between the real and virtual worlds. All the procedures must be done in real time. This paper proposes a method for augmented reality with a stereo vision sensor and a video see-through head-mounted display (HMD). It can synchronize the display timing between the virtual and real worlds so that the alignment error is reduced. The method calculates camera parameters from three markers in image sequences captured by a pair of stereo cameras mounted on the HMD. In addition, it estimates the real-world depth from a pair of stereo images in order to generate a composed image maintaining consistent occlusions between real and virtual objects. The depth estimation region is efficiently limited by calculating the position of the virtual object by using the camera parameters. Finally, we have developed a video see-through augmented reality system which mainly consists of a pair of stereo cameras mounted on the HMD and a standard graphics workstation. The feasibility of the system has been successfully demonstrated with experiments.

We present a new free-form interactive modeling technique based on the metaphor of clay work. This paper discusses design issues and an immersive modeling system which enables a user to design intuitively and interactively 3D solid objects with curved surfaces by using one's finger. Shape deformation is expressed by simple formulas without complex calculation because of skeletal implicit surfaces employed to represent smooth free-form surfaces. A polygonization algorithm that generates polygonal representation from implicit surfaces is developed to reduce the time required for rendering curved surfaces, since conventional graphics hardware is optimized for displaying polygons. The prototype system has shown that a user can design 3D solid objects composed of curved surfaces in a short time by deforming objects intuitively using one's finger in real time.

This paper proposes a new self-localization method using an omnidirectional image sensor which can observe a surrounding environment with 360-degree of view. The method extracts information which is identical for the position of a sensor and invariant against the rotation of the sensor by generating an autocorrelation image from an observed omnidirectional image. The location of the sensor is estimated by evaluating the similarity among the autocorrelation image of an observed image and stored autocorrelation images. The similarity of autocorrelation images is evaluated in low dimensional eigenspaces generated with stored autocorrelation images. We have conducted experiments with real images and examined the performance of the proposed method. The results show that accurate and robust estimation of the sensor's position is possible with our method.

In this paper, we propose a new method for detection and tracking of moving objects from a moving camera image sequence using robust statistics and active contour models. We assume that the apparent background motion between two consecutive image frames can be approximated by affine transformation. In order to register the static background, we estimate affine transformation parameters using LMedS (least median of squares) method which is a kind of robust statistics. Split-and-merge contour models are employed for tracking multiple moving objects which have been recently proposed by the authors. Image energy of contour models is defined based on the image which is obtained by subtracting the previous frame transformed with estimated affine parameters from the current frame. We have implemented the method on an image processing system which consists of DSP boards for real-time tracking of moving objects from a moving camera image sequence.

This study proposes a new factorization method for shape and motion recovery. In the past, the fourth greatest singular value of the measurement matrix was ignored. But when noise is large enough so that the fourth greatest singular value can not be ignored, it would be difficult to get reliable results by using the traditional factorization method. In order to acquire reliable results, We start with adopting an orthogonalization method to find a matrix which is composed of three mutually orthogonal vectors. By using this matrix, another matrix can be obtained. Then, the two expected matrices which represent shape of object and motion of camera/object, can be obtained through normalization. This study also conducts several experiments to discuss the feasibility of the proposed method.

We have developed a high-resolution omnidirectional stereo imaging sensor that can take images at video-rate. The sensor system takes an omnidirectional view by a component constructed of six cameras and a hexagonal pyramidal mirror and acquires stereo views by symmetrically connecting two sensor components. The paper describes a method of generating stereo panoramic images by using our sensor. First, the sensor system is calibrated; that is, twelve cameras are correctly aligned with pyramidal mirrors and the Tsai's method restores the radial distortion of each camera image. Stereo panoramic images are then computed by registering the camera images captured at the same time.

Describes a prototype of an augmented reality system using vision-based registration. In order to build an augmented reality system with video see-through image composition, camera parameters for generating virtual objects must be obtained at video-rate. The system estimates camera parameters from four known markers in an image sequence captured by a small CCD camera mounted on a HMD (head mounted display). Virtual objects are overlaid upon a captured image sequence in real-time.

This paper describes a visual surveillance and monitoring system which is based on omnidirectional imaging and view-dependent image generation from omnidirectional video streams. While conventional visual surveillance and monitoring systems usually consist of either a number of fixed regular cameras or a mechanically controlled camera, the proposed system has a single omnidirectional video camera using a hyperboloidal mirror. This approach has an advantage of less latency in looking around a large field of view. In a prototype system developed, the viewing direction is determined by viewers' head tracking, by using a mouse, or by moving object trading in the omnidirectional image.

This paper describes a new method for extracting facial components from a color image using cooperative active nets with global constraints. We utilize an active net model which is a region extraction method based on energy minimization principle. Each net deforms with its own energy being minimized and the position of the nets are controlled by minimizing an additional energy defined by global constraints on their placement. Our method has been experimentally shown to be robust to variations of facial size, position and rotation.

Registration and integration of measured data of real objects are becoming important in 3D modeling for computer graphics and computer-aided design. We propose a new algorithm of registration and integration of multiple range images for producing a geometric object surface model. The registration algorithm determines a set of rigid motion parameters that register a range image to a given mesh-based geometric model. The algorithm is an integration of the iterative closest point algorithm with the least median of squares estimator. After registration, points in the input range are classified into inliers and outliers according to registration errors between each data point and the model. The outliers are appended to the surface model to be used by registration with the following range images. The parts classified as inlier by at least one registration result is segmented out to be integrated. This process consisting of registration and integration is iterated until all views are integrated. We successfully experimented with the proposed method on real range image sequences taken by a rangefinder. The method does not need any preliminary processes.

The style of cryptography in which important data are embedded into image data not to be noticed by outsiders is called video steganography. This paper proposes a new method for embedding data into pictorial images. In this method, the amount of locally embedded data is adaptively varied according to the local characteristics of the original image, and more data will be embedded into the image with less distortion. Using the discrete cosine transform (DCT), data are mainly embedded into high frequency components of the image, which have little influence on the quality of the image. The amount of embedded data is varied depending on DCT components of adjacent areas not to weaken edge lines in the image. We also show the effectiveness of our method by subjective evaluation.

This paper describes a synthesis-by-analysis approach using cylindrical range images for producing human facial images with realistic expressions. First view-independent representations of 3D locations of facial feature points are obtained by using an object-centered coordinate system defined on a face. Then we quantify facial feature points locations for the neutral expression, and six primary expressions: "anger", "disgust", "surprise", "fear", "happiness" and "sadness" Applying an image warping technique to both range and texture images, we finally generate 3D facial expression images from neutral expression images using motion vectors of facial feature points.

This paper describes a case study of building a prototype of an immersive three dimensional (3-D) modeler which supports simple two-handed operations. Designing 3-D objects in a virtual environment has a number of advantages for 3-D geometry creation over designing with traditional computer aided design (CAD) tools. In order to enhance the human-computer interaction in a virtual workspace, two-handed spatial input has been incorporated into a few 3-D designing applications. However, existing 3-D designing tools do not utilize two handed interaction for enhancing the interface sufficiently. Our prototype immersive modeler, VLEGO, employs some features of toy blocks to give flexible two-handed interaction for 3-D design. Features of VLEGO can be summarized as follows: Firstly, VLEGO supports various two-handed operations and hence it makes design environment intuitive and efficient. Secondly, possible location and orientation of primitives are discretely limited so that the user can arrange objects accurately with ease. Finally, the system automatically avoids collisions among primitives and adjusts their positions. As a result, precise design of 3-D objects can be achieved easily by using a set of two-handed operations in intuitive way. This paper describes the design and implementation of VLEGO as well as an experiment for examining the effectiveness of two-handed interaction.

This paper describes a method of establishing dense matching of two views with large displacements. The problem addressed is formulated as the minimization of an energy functional that combines a similarity term and a smoothness term. The minimization of the energy functional reduces to solving a large system of nonlinear equations which is a discretized version of the Euler-Lagrange equation for the energy minimization problem at each image point. A dense displacement map is computed by solving the system of equations using a coarse-to-fine approach. The method has been successfully experimented with two applications: (1) computing binocular disparities from stereo pair images and (2) computing a dense displacement vector field (optical flow) from two views in a time-varying image sequence.>

Describes a method for tracking a deformable object in a time-varying image sequence and analyzing the motion of the object. The approach is based upon modeling the boundary of the object as a new active contour model defined by an energy functional in terms of intra- and inter-frame constraints, and then finding the object boundary in the temporal images by minimizing the energy functional using dynamic programming in multiple scales. Motion analysis is carried out by analyzing some characteristic points which are extracted as local curvature extrema along the contour. The proposed method has been successfully applied to the analysis of a slug (Limax flavus) in motion with smooth deformation, in the study of the mechanisms of learning and memory formation in slugs.>

This paper describes a method of surface reconstruction from stereo images. The problem addressed is formulated as the minimization of an energy functional that integrates a similarity term and a smoothness term. The minimization reduces to solving a system of nonlinear equations. A disparity map is obtained by solving the system of equations through scale hierarchy. The proposed method can compute a dense disparity map directly from stereo images preserving discontinuities.>

Describes a method for recovering volumetric shape descriptions of 3D objects from range data. The models of shape primitives are superquadrics with a global deformation of tapering. The authors first define a new error-of-fit measure for fitting superquadrics to range data. The measure evaluates the difference in both depth and surface orientation between input 3D data and the model surfaces. Simulated annealing is then employed for finding a global minimum of the error-of-fit over all possible values of model parameters including translation, rotation and tapering.>

An approach to the problem of finding surfaces of revolution in range data is proposed. It is based on the fact that at least one of two focal surfaces for a surface of revolution degenerates into the axis of rotation. First, the centers of principal curvature are obtained for each point in the range image. Then, the Hough transform is used to detect the axes of rotation by finding the centers which lie on straight lines in space. Finally, the solid of revolution is volumetrically determined by estimating the cross-sectional radius function along each axis. It is concluded that the method can be used for describing and recognizing 3-D objects partially constructed of an unrestricted class of surfaces of revolution which are partially visible in a range image.>

The authors describe a hybrid approach to the problem of image segmentation in range data analysis, where hybrid refers to a combination of both region- and edge-based considerations. The range image of 3-D objects is divided into surface primitives which are homogeneous in their intrinsic differential geometric properties and do not contain discontinuities in either depth of surface orientation. The method is based on the computation of partial derivatives, obtained by a selective local biquadratic surface fit. Then, by computing the Gaussian and mean curvatures, an initial region-gased segmentation is obtained in the form of a curvature sign map. Two additional initial edge-based segmentations are also computed from the partial derivatives and depth values, namely, jump and roof-edge maps. The three image maps are then combined to produce the final segmentation. Experimental results obtained for both synthetic and real range data of polyhedral and curved objects are given.>