This video demonstrates a mixed reality (MR) environment which is constructed for development of autonomous behaviors of robots. Many kinds of functions are required to be integrated for realizing an autonomous behavior. For example, autonomous navigation of humanoid robots needs functions, such as, recognition of environment, localization and mapping, path planning, gait planning, dynamically stable biped walking pattern generation, and sensor feedback stabilization of walking. Technologies to realize each function are well investigated by many research works. However, another effort is required for constructing an autonomous behavior by integrating those functions. We demonstrate a MR environment in which internal status of a robot, such as, sensor status, recognition results, planning results, and motion control parameters, can be projected to the environment and its body. We can understand intuitively how each function works as a part of total system in the real environment by using the proposed system, and it helps solving the integration problems. The overview of the system, projection of each internal status, and the application to an autonomous locomotion experiment are presented in the video clip.

One of the problems in developing a humanoid robot is caused by the fact that intermediate results, such as what the robot perceives the environment, and how it plans its moving path are hard to be observed online in the physical environment. What developers can see is only the behavior. Therefore, they usually investigate logged data afterwards, to analyze how well each component worked, or which component was wrong in the total system. In this paper, we present a novel environment for robot development, in which intermediate results of the system are overlaid on physical space using mixed reality technology. Real-time observation enables the developers to see intuitively, in what situation the specific intermediate results are generated, and to understand how results of a component affected the total system. This feature makes the development efficient and precise. This environment also gives a human-robot interface that shows the robot internal state intuitively, not only in development, but also in operation.

We propose a hybrid camera pose estimation method using an inclination sensor value and correspondence-free line segments. In this method, possible azimuths of the camera pose are hypothesized by a voting method under an inclination constraint. Then some camera positions for each possible azimuth are calculated based on the detected line segments that affirmatively voted for the azimuth. Finally, the most consistent one is selected as the camera pose out of the multiple sets of the camera positions and azimuths. Unlike many other tracking methods, our method does not use past information but rather estimates the camera pose using only present information. This feature is useful for an initialization measure of registration in augmented reality (AR) systems. This paper describes the details of the method and shows its effectiveness with experiments in which the method is actually used in an AR application.

This paper describes an evaluation system using an industrial robot, constructed for the purpose of evaluating registration technology for Mixed Reality. In this evaluation system, the tip of the robot arm plays the role of the user's head, where a head- mounted display is mounted. By using an industrial robot, we can obtain the ground truth of the camera pose with a high level of accuracy and robustness. Additionally, we have the ability to play back the same specified operations repeatedly under identical conditions. In addition to the system implementation, we propose evaluation methods for motion robustness, relative orientation robustness, relative distance robustness, jitter, and an overall evaluation. We verify the validity of this system through some experiments.

This work describes a new hybrid vision-based registration method with an inclination sensor. In the method, a camera is tracked by solving linear equations under inclination constraint. Linear operations are faster than the nonlinear optimization process, but the output does not usually satisfy the orthonormality constraint. On the contrary, the proposed method calculates the camera position and azimuth directly, thus the result satisfies the orthonormality constraint. Many hybrid approaches using inertia sensors have been proposed for AR/MR. However, such methods still depend on vision-based methods in initialization processes. On the other hand, the proposed method is totally hybrid in that the inclination measured by the sensor is always incorporated in pose calculation process as well as vision information. The method can be used in initialization process of conventional hybrid methods as well as it can be used as an independent registration method. The proposed method can be applied to not only inside-out-style camera tracking but also outside-in-style object tracking.

This paper describes a calibration method of markers which are used for registration in MR applications. There have been many vision-based approaches proposed as registration methods in MR. When multiple markers are utilized in a vision-based method, it is necessary that the geometric information of the marker arrangement such as their positions and orientations be known in advance. In this paper, we propose a hybrid method combining the "bundle adjustment method," which is a photogrammetric technique that calculates the geometric information from a set of images, with some constraints on the marker arrangement which are obtained a priori (e.g. the multiple markers are located on a single plane). After considering marker arrangements seen in many MR systems, we summarize some constraints seen in these arrangements. Then, we explain the basic framework of this method as well as a solution method under some practical constraints. We further describe several experiments and their results in order to show the effectiveness of this method.

This paper describes a new head tracking method utilizing a gyroscope mounted on a head-mounted display (HMD) and a bird's-eye view camera that observes the HMD from a fixed third-person viewpoint. Furthermore, we propose an extension of this method to hybrid registration, combining it with user's view cameras. The HMD is equipped with a gyroscope and a marker. The gyroscope measures the orientation of the user s view camera so that the number of pose parameters to be solved can be reduced. The other parameters are to be estimated by the bird's-eye view camera that observes the marker. This method is an improvement over the conventional outside-in-style vision-based tracker method, which only uses visual information. Hence, it can be thought of as an alternative to a physical head tracker such as a magnetic sensor and to an inside-out-style vision-based tracker. In addition to theoretical discussions, this paper demonstrates the effectiveness of our methods by experiments. We also propose methods for calibrating the gyroscope and the marker on HMD, which are essential in implementing the tracking method.

This paper describes new vision-based registration methods utilizing not only cameras on a user's head-mounted display but also a bird's-eye view camera that observes the user from an objective viewpoint. Two new methods, the line constraint method (LCM) and global error minimization method (GEM), are proposed. The former method reduces the number of unknown parameters concerning the user's viewpoint by restricting it to be on the line of sight from the bird's-eye view. The other method minimizes the sum of errors, which is the sum of the distance between the fiducials on the view and the calculated positions of them based on the current viewing parameters, for both the user's view and the bird's-eye view. The methods proposed here reduce the number of points that should be observed from the user's viewpoint for registration, thus improving the stability. In addition to theoretical discussions, this paper demonstrates the effectiveness of our methods by experiments in comparison with methods that use only a user's view camera or a bird's-eye view camera.

This paper describes a platform package, called "MR Platform," which we have been implementing for research and development of augmented reality technology and applications. This package includes a parallax-less stereo video see-through HMD and a software development kit (SDK) for a Linux PC environment. The SDK is composed of a C++ class library for making runtime MR applications and related utilities such as a camera calibration tool. By using the SDK, the following functions are available: capturing video, handling a six degree-of-freedom (DOF) sensor, image processing such as color detection, estimating head position and orientation, displaying the real world image, and calibrating sensor placement and camera parameters of two cameras mounted on the HMD.