In this paper, we describe the real-time and precise self collision detection system that does not reduce the number of polygons and checks more than 100 collision pairs in real-time by using AABB based collision detection libraries. Previous researches on collision detection of humanoid robots which reduce collision pairs or simplify a shape of a robot has disadvantages such as increasing the dangerousness or decreasing range of movement. However our self collision detection system uses detailed geometric model and collision pairs as many as possible. We have experimentally evaluated collision detection libraries on a real-time self collision detection application of a humanoid robot. This experiment suggests that AABBs based method is much faster than conventional OBBs based method. Finally, we demonstrated real-time collision detection and avoidance function that automatically stops entire motion if self collision occurs using HRP2 humanoid robot.

This paper describes a 32 /spl times/ 32 matrix scan type high-speed pressure sensor for the feet of humanoid robots that has 1 kHz sampling rate. This sensor has matrix scan circuit. The matrix scan method has a problem of interference by bypass current. To resolve this problem, we suggest a novel method using a very thin conductive rubber. We adopted a very thin (0.6 mm) force sensing conductive rubber sheet for high speed sensing. Each sensing area is 4.2 /spl times/ 7.0 mm and can measure vertical force of approximately 0.25-20 N. Walking cycle of humanoid robot as well as human being is about 0.4-0.8 s and dual leg phase is about 0.1-0.15 s. The target of the sensor is biped walk stabilization so that high-speed input is important. Matrix scan type circuit is connected to sensor, and the system runs 1 kHz with 14 bit resolution at 4.2 /spl times/ 7.0 mm grid for 32 /spl times/ 32 points, and the sensor size is the same as humanoid robot foot 135 /spl times/ 228 mm, The system is running high-speed because of the very thin conductive rubber and simultaneous measurement. The sensor system, a novel scan method, and evaluation results are described.

This paper describes an online system for footstep planning using a 3D map reconstructed by visual odometry. This system consists of two key components: 3D reconstruction via visual odometry from a stereo image sequence to obtain a dense local world model, and a footstep planner for biped robots using the reconstructed 3D map. Visual odometry is a method to connect 3D image sequences to obtain 6DOF camera motion and dense 3D environment information. The method described in this paper consists of three components: stereo depth map calculation, 3D flow calculation from tracking raw image features, and 6DOF camera motion estimation from RANSAC. Using the resulting 3D data, an optimal sequence of footstep locations is planned. The footstep planner is provided a height map of the terrain and a discrete set of possible footstep motions. The planner then evaluates footstep locations for viability using a collection of heuristic metrics designed to encode the relative safety, effort required, and overall motion complexity. Finally, we implemented this system on the humanoid robot H7. A local 3D map is reconstructed using visual odometry at about 10 Hz and the footstep planner replans at intervals of four steps. The robot walked across a floor, avoiding obstacles and reaching the goal.

This paper describes the design and development of system software for humanoid robots such that researchers who specialize not only in biped walking but also in various fields are able to use humanoid robots as a research tool. For this purpose, the system for a humanoid must integrate and organize each subsystem such as control, recognition, dialogue, planning and so on, and it must provide efficient full-body motion control by specifying fewer degrees of freedom than all joints. Our system design provides a common interface among subsystems by implementing each function as a method call through a three-dimensional model of the robot for good integration, and it also provides a motion planning technique based full-body posture sequence and walking pattern generation. Finally, we show integrated behavior experiments with vision, planning and motion control using the developed system software for a life-sized humanoid robot, HRP2.

This paper proposes the mathematical model of deformation for gel robots and develops the pattern formation theory. The robots are made of surfactant-driven ionic polymer gel in constant electric fields, which is a typical electroactive polymer gel containing poly 2-acrylamido-2-methylpropane sulfonic acid (PAMPS). A beam of gel in uniform electric fields develops wave forms through penetration of the surfactant solution. The model is to be built on the hypothesis of adsorption-induced deformation. The mechanism of wave-shape pattern formation is then analyzed utilizing the model. The results of this study provide the foundation to develop deformable machines with virtually infinite degrees of freedom.

In this paper we present "hand-centered" motion generation and control of whole-body motion for a humanoid robot, which enable the robot to carry out reaching and walking simultaneously. Hand-centered whole-body motion control is defined as a real-time control scheme in which the hand motion (input) leads to the whole-body motion (output). The hand trajectory is described in the absolute coordinate system (world frame) and the torso trajectory follows the hand trajectory. This method enables operation of the hand of a walking humanoid robot, and autonomous generation and control of the whole-body motion based on the hand velocity vector using sensor feedback. We have implemented the method as a hand-centered whole-body motion control system on the existing humanoid robot system "H7". Some experimental results including both reaching and walking motions are shown to demonstrate the effectiveness of the proposed method.

In this paper, we focused on concepts of behavior modifier. We aim to build a system, which acquires concepts of behavior modifier and applies it to motion generation of robots. For a form of motion representation, an existing research achievement, which statistically abstracts motion with known sets of motion examples and represents motions as points in a space, was adopted. Our system uses tendency of motions with adverbial modifier in the space to represent concepts of behavior modifier. The system has a simple representation form and has interactiveness what has potential advantage of ability to be applied to various motion concepts.

In this paper, we propose an approach where robots store shared experiences between human and the robots, and also show that the shared experiences act as an important role not only for the coexistence of the human and robots but also for the realization of the supporting ability of the robots. Enormous amount of shared experiences are effective for the understanding of users under uncertain and incomplete conditions, as well as communication among human. We propose the importance of the shared experiences for real world intelligence, and show a development research of the infrastructure for the storage of the share experiences and application method.

We propose a concept of reinforceable-muscle humanoid with hyper parallel muscle-tendon systems. By the facilitation of changing the assignment of actuators, we can easily strengthen a specific part of robot body, while changing the arrangement of actuators of current humanoid robots is quite difficult. By developing muscle units in each of which a motor and sensors are integrated, and by adopting musculo-skeletal structure, the rearrangeability of muscles has been realized. This paper describes the concept, prototype design and implementation of the reinforceable muscle humanoid, and the evaluating experiment by a musculo-skeletal humanoid is presented.

We propose the portable situation reporting system by a small robot for daily life, which demands the rapidly system reconstitution for daily life support. In this paper, we established a system by an attachment mechanism assisted the robot system based on device-distributed approach and the patterned processing by the tree structuring of sensor information processing. The attachment mechanism facilitates the hardware system reconstitution and the patterned processing simplifies the software system restructuring. We evaluated the reconfigurable system not to re-write programs and judged the effectiveness of the system by the experiments.

In this paper, we describe a planner for a humanoid robot that is capable of finding a path in an environment with movable objects, whereas previous motion planner only deals with an environment with fixed objects. We address an environment manipulation problem for a humanoid robot that finds a walking path from the given start location to the goal location while displacing obstructing objects on the walking path. This problem requires more complex configuration space than previous researches using a mobile robot especially in a manipulation phase, since a humanoid robot has many degrees of freedom in its arm than a forklift type robot. Our approach is to build environment manipulation task graph that decompose the given task into subtasks which are solved using navigation path planner or whole body motion planner. We also propose a standing location search and a displacing obstacle location search for connecting subtasks. Efficient method to solve manipulation planning that relies on whole body inverse kinematics and motion planning technology is also shown. Finally, we show experimental results in an environment with movable objects such as chairs and trash boxes. The planner finds an action sequence consists of walking paths and manipulating obstructing objects to walk from the start position to the goal position.

Since an environment or body states of robots are very changeable, robotic imitation systems should have ability to develop their behaviors by themselves. For the development of the imitation behaviors, we assume that attention during imitation is the key information. In order to realize such imitation systems with evolving ability, the idea of reinforcement learning system based on the attention structure during imitation has been presented in this paper. First, for describing the attention during imitation behaviors, we define the term 'sensor-action attention pair' as the pair of the most important sensor information and the focused body parts during that behavior. Second, we introduce R-learning, the reinforcement learning method for continual tasks such as imitation behaviors. Third, the method to design the state-action space and the reward function based on the sensor-action attention pair is proposed. At last, for the confirmation of the function of the proposed imitation behavior system, we have done some experiments using actual humanoid Kenta. In those experiments, Kenta can develop imitation behavior that imitates the hand position of the human.

We present an integrated humanoid locomotion and online terrain modeling system using stereo vision. From a 3D depth map, a 2.5D probabilistic description of the nearby terrain is generated. The depth map is calculated from a pair of stereo camera images, correlation-based localization is performed, and candidate planar walking surfaces are extracted. The results are used to update a probabilistic map of the terrain, which is input to an online footstep planning system. Experimental results are shown using the humanoid robot H7, which was designed as a research platform for intelligent humanoid robotics.

We present an online algorithm for planning sequences of footstep locations that encode goal-directed navigation strategies for humanoid robots. Planning footsteps is more general than most existing navigation methods designed for wheeled robots, since the options of stepping over or upon obstacles in a cluttered terrain are available. Given a discrete set of plausible footstep locations, a forward dynamic programming approach is used to compute a footstep sequence to a specified goal location in the environment. Heuristics designed to minimize the number and complexity of the step motions are used to encode cost functions used for searching a footstep transition graph. If successful, the planner returns an optimal sequence of footstep locations according to the cost functions and plausible sets of footstep locations defined. We show results from an experimental implementation of the algorithm running on the H7 humanoid robot. Using a stereo vision system to sense obstacles in the immediate environment and identify a target goal location, the robot updates the current optimal footstep sequence to the goal from its present location.

Behavior imitation ability will be a key technology for future human friendly robots. In order to understand the principles and mechanisms of imitation, we take a synthetic cognitive developmental approach, starting with minimum components and create a system that can learn to imitate others. We developed a visuo-motor neural learning system which consists of orientation selective visual movement representation, distributed arm movement representation, and a high-dimensional temporal sequence learning mechanism. The vision and the movement representations model the findings in primate brain, i.e. macaque area MT(or human area V5) and the primary motor area. The learning mechanism is inspired by the finding that there are excessive connections in neonate brain. As our robot explores the visuo-motor self movement patterns, it learns coherent patterns as high-dimensional trajectory attractors. After the learning, a human comes in front of the robot showing arm movements which are similar to the ones in self learning. Although the robot has never seen or programmed to interpret human arm movement, and the detail of visual stimuli are very different, the robot identifies some of the patterns as similar to those in self learning, and responded by generating the previously learned arm movement. In other words, the robot exhibits early imitation ability based on self exploratory learning.

It is very important and hard to develop motions and behaviors for robots that have complex body and numerous actuators and sensors. The more complex a humanoid's body becomes, the more important the developing environment of the motions and behaviors will be. We have built a fully muscle-driven flexible-spine humanoid named "Kenta", which has 96 actuators and more than 400 sensors including tension sensors, posture sensors, vision sensors, and so on. This paper describes the developing environment of motions and behaviors for the robots like Kenta. A visualizable geometric model, a simulation environment transparent to the real environment, sample procedures of the motion development, and, as an example, realization of swing behavior by Kenta are presented.

We present a humanoid walking control system that generates body trajectories to follow a given desired motion online. A layered software and control architecture is used to aggregate system components and provide a framework for high-level autonomous locomotion behaviors. Walking characteristics such as desired torso movements, upper body posture and step cycles, can be specified and used to generate stable whole-body walking trajectories online. The basic architecture consists of 4 layers: footstep planning, trajectory generation, trajectory modification from sensor feedback, and joint servo control. In order to test the complete system, we implemented an experimental top-level behavior that enables the robot to track and follow a moving target based on stereo vision feedback. We explain the overall system design and components, and present results based on our current implementation using the humanoid robot 'H7'.

This paper formulates and solves the inverse dynamics problem of deformable robots made entirely of electro-active polymer gel. One of the primary difficulties with deformable robots is that they have conceptually infinite degrees of freedom. We solve this problem through the selection of an essential point to generate a desired motion. The problem is then reduced to trajectory control of a point on the robot. We have proposed dynamic models of electro-active polymers system and derived a variety of motions by applying either spatially or time varying electric fields. However, the motion control problem has not yet been investigated. We show a procedure to realize an inversion (turning over) motion of a starfish-shaped gel robots by applying both spatially varying and time alternating electric fields. This work takes the first step towards motion control of deformable robots.

This paper describes an experimental stereo vision based motion planning system for humanoid robots. The goal is to automatically generate arm trajectories that avoid obstacles in unknown environments from high-level task commands. Our system consists of three components: 1) environment sensing using stereo vision with disparity map generation and on-line consistency checking, 2) probabilistic mesh modeling in order to accumulate continuous vision input, and 3) motion planning for the robot arm using RRTs (rapidly exploring random trees). We demonstrate results from experiments using an implementation designed for the humanoid robot H7.

In this paper, the method of motion planning for the metamorphic robot which can change the rigidity and the shape of its body is described. When making motions of the metamorphic robot, we can use not only the degrees of freedom in the joints but also the deformation of the links. In making use of the deformation by the length of the shortest path, this connects both ends of the deformed link. Balance constraints which depend on the friction between the body and the environment need to be considered in order to maintain the stability of the motions. In this time, we check the stability by computing contact forces considering the moving direction of the contacts. The motion planning is presented using Linkrobot which is made of metamorphic links and can deform its whole body.

To realize humanoid robots in unknown environment, sensor based navigation system is required as one of an essential function. This paper describes vision-based navigation system for humanoid robots, which has following features: 1) To recognize floor regions from a view of vision of a humanoid robot in unknown environment, we utilized existing technique called Plane Segment Finder, which is able to extract arbitrary planner surface regions from depth image. 2) Path planning for wheeled robots usually models a robot as a 2D circle, however path planning system for humanoid robot requires capable of modeling a robot as a 3D cylinder model, convex hull model, rigid model and so on, according to a situation such as a robot carries a large object or a robot opens its arms. Finally, we show a humanoid robot HOAP-1 with enhanced stereo vision system for navigation task and a result of path planning using generated local map through stereo vision system which uses real images as an input.

We present an approach to self-collision detection suitable for complex articulated robots such as humanoids. Preventing self-collisions is vital for the safe operation of robots that generate body trajectories online. Our approach uses a fast distance determination method for convex polyhedra in order to conservatively guarantee that a given trajectory is free of self-collision. Experimental results using an online joystick control application for the humanoid robot "H7" demonstrate the feasibility and effectiveness of the method.

Addresses the extension of a humanoid's action capability by attaching toe joints. The effectiveness of toe joints is discussed in three aspects. One is utilizing it to speed up the walking, another is using it to enable a humanoid to go up higher steps, and the other is using it to whole-body action in which knees are contacting the ground. Feet with toe joints are developed for humanoid 'H6'. An experiment of the wholebody action in which knees are contacting the ground is carried out to show the usefulness of toe joints for such actions. Then the walking pattern generation system is extended to use toe joints. Using this extended system maximum speed of knee joints can be reduced at the same walking speed, and 80% faster walking speed is achieved on humanoid 'H6'.

This paper describes a design of six-axis force sensor that measures ground reaction force of human or humanoid robot. The key concept is parallel support mechanisms, that allow large torques and forces which are caused when foot is hitting to the environment. Basic concept and design of parallel support mechanisms are denoted. Finally ground reaction force measurement system for human walking, and application to humanoid robot walking are described.

This paper proposes an inverse kinematic model for deformable robots made entirely of electro-active polymer gel. The required method is to control a higher degrees of freedom than numbers of input. We (2000, 2001) have been proposed a kinematic and dynamic model of electro-active polymer system and derived a variety of motions of gel robots by applying spatially varying electric fields. However, inverse kinematic model and the method of applying time alternating electric fields have not been investigated. We challenge the tip control of gel manipulator by applying spatially uniform but time varying electric field. We show the procedure to control the tip position of a gel manipulator by dynamically and slightly changing its whole configuration. Our work is a first step towards the shape control of gel robots.

A humanoid robot is expected to be a rational form of machine to act in the real human environment and support people through interaction with them. Current humanoid robots, however, lack in adaptability, agility, or high-mobility enough to meet the expectations. In order to enhance high-mobility, the humanoid motion should be generated in real-time in accordance with the dynamics, which commonly requires a large amount of computation and has not been implemented so far. We have developed a real-time motion generation method that controls the center of gravity (COG) by indirect manipulation of the zero moment point (ZMP). The real-time response of the method provides humanoid robots with high-mobility. In the paper, the algorithm is presented. It consists of four parts, namely, the referential ZMP planning, the ZMP manipulation, the COG velocity decomposition to joint angles, and local control of joint angles. An advantage of the algorithm lies in its applicability to humanoids with a lot of degrees of freedom. The effectiveness of the proposed method is verified by computer simulations.

This paper describes the design of an integrated system for humanoid robotics that consists of three key components, 3D vision, motion planning and bipedal control. Layered system design is adopted to achieve concurrency as well as small latency. Implementation using our humanoid type robot H7 and experiments with this architecture are described. The H7 is expected to be a common test-bed in experiments and discussion for various aspects of intelligent humanoid robotics.

We are trying to realize a humanoid which has flexibility. If humanoids have a flexible structure, safety and posture variety can be achieved. We focus on the role of the human spine and muscle-driven system. By having a flexible spine, a humanoid will have safety and many degrees of freedom to realize a variety of postures. By driving joints by tension-controllable tendons, flexibility of the joints can be controlled. We developed a whole-body tendon-driven flexible-spine humanoid named "Kenta". This paper describes the design and control of Kenta, focusing on the design of the spine. The spine consists of ten joints, vertebrae and rubber disks, ribs, and forty muscles equipped with tension sensors. We also propose control methods of the spine. One uses a geometric virtual robot model and another is based on direct teaching. Using these methods, some whole-body motions are presented.

In this paper, a robot that can change the rigidity and shape of its body is called "metamorphic robot". So far robots have rigid bodies and have their degrees of freedom only in the joints. The metamorphic robot can change their bodies fit to the environment by deforming the limbs between the joints. To realize this robot, the softening deformable structure is developed. The structure is made of a low melting point alloy and becomes soft and deformable through the phase change induced by heating and becomes rigid by cooling. In the soft-deformable state, the robot can change the shape by pressing against objects. On the other hand, in the rigid state it is rigid enough to support the robot's body weight keeping the deformation.

This paper presents an efficient online method to generate humanoid walking motions that satisfy desired upper body trajectories while simultaneously carrying objects. A fast motion pattern generation technique that follows the desired ZMP is adopted. In order to satisfy the control input given online, subsequent motion patterns are updated and connected in a stable manner to the old ones while executing. During the creation of motion trajectories online, the commanded motion parameters are checked and modified automatically considering the performance limitations of the hardware. As an example application, we have implemented a one step cycle control system on the Humanoid H7. Experiments controlling the upper body motion and walking direction using a joystick interface are explained to demonstrate the validity of the proposed method.

This paper describes a rapid development system for humanoid vision-based behavior consisting of both real and virtual robots in a simulation environment. Previous robotics simulators were limited to verify dynamic motions such as walking, or to plan or learn in a simple environment. However, our system is able to simulate vision based behavior, i.e. motion and perception behavior, of a robot as a whole, since it can simulate both dynamics and collisions in the environment. We regard the real-time aspect of the simulation as important so that so that users can develop vision based behavior rapidly and efficiently. We employ a Common Interface API to share behavior software between real and virtual robots. Moreover, visual processing functions such as color extraction and depth map generation are available. As a result, vision based behavior consisting of local map generation, planning and navigation of the humanoid in simulation is presented. We also show that software developed in the simulation environment is applicable to real robots.

This paper describes a design of remote operation interface for humanoid robot with following three functions: (1) Body DOFs control interface through 3D robot model in a virtual environment, and it has two types of online stabilizing software for maintaining the body balance. (2) Environmental recognition display interface for sensors, sound and 3D vision. (3) Voice and face recognition interface for interacting with operator and a human in front of the robot body. Then, implementation and experiments on our humanoid robot H6 are denoted.

The H6 humanoid robot has been developed as a platform for the research on perception-action coupling in intelligent behaviour of humanoid type robots. The H6 features: 1) a body which has enough DOFs and each joint has enough torque for full body motion; 2) a PC/AT compatible high-performance on-board computer which is controlled by RT-Linux so that from low-level to high-level control is achieved simultaneously; 3) self-contained and connected to a network via radio Ethernet; and 4) dynamic walking trajectory generation, motion planning and 3D vision functions are available. The H6 can be used as a common test-bed in experiment for various aspects of intelligent humanoid robotics.

In this paper, the developmental software environment for humanoids is introduced. This environment has the following features: 1) a hardware concealment function which saves the trouble with hardware improvements; 2) an incremental expansion function which supports a long-term development by many developers; and 3) a centralised management of body information which enables the robots to share software among different bodies. These features are realized by the plug-in architecture, the universal robot model and its compilers. Using plug-ins enables one to accumulate functions developed by different developers and to switch computer interfaces of the robot body. The universal robot model and compilers promote the use of external software and share software modules among different bodies.

We present an approach to path planning for humanoid robots that computes dynamically-stable, collision-free trajectories from full-body posture goals. Given a geometric model of the environment and a statically-stable desired posture, we search the configuration space of the robot for a collision-free path that simultaneously satisfies dynamic balance constraints. We adapt existing randomized path planning techniques by imposing balance constraints on incremental search motions in order to maintain the overall dynamic stability of the final path. A dynamics filtering function that constrains the ZMP (zero moment point) trajectory is used as a post-processing step to transform statically-stable, collision-free paths into dynamically-stable, collision-free trajectories for the entire body. Although we have focused our experiments on biped robots with a humanoid shape, the method generally applies to any robot subject to balance constraints (legged or not). The algorithm is presented along with computed examples using the humanoid robot "H6".

This paper introduces and describes a novel method which enables the online generation of humanoid walking patterns that follow desired footprint locations. Online generation is realized by the dynamically stable mixture and connection of pre-designed motions. Characteristics of zero moment point (ZMP) are utilized in order to maintain the overall dynamic stability of "mixed" motions. Experiments using an online pattern generation software system and online walking control joystick interface for the humanoid robot H6 are introduced to show the validity of the method.

This paper describes the development of a plane segment finder, which is able to detect three-dimensional planar surfaces from input images in real-time. We propose an algorithm for detecting plane segments, that includes: 1) plane segment candidate extraction using the 3D Hough transform from depth map information; and 2) fitting the plane segment candidates to the depth map in order to detect the partial plane segment, since the extracted plane segment candidates are general planes, with no boundary. To achieve real-time plane segment finding system, we apply: 1) the recursive correlation method for depth map generation; and 2) the randomized Hough transform for plane segment extraction. Finally, experimental results using an implementation of our system along with a humanoid robot are shown.

This paper shows walking human avoidance and detection behavior of a mobile robot using the 3D depth flow. The 3D depth flow proposed is able to measure the 3D motion vector of every pixels between two time sequential images. First, a definition of the 3D depth flow and a simple 3D depth flow calculation method are presented. Then an implementation of the real-time 3D depth flow generation system using a standard PC is described, and the experimental results are given. Finally, as an application, the walking human detection and avoidance task using a mobile robot in real environments is shown.

This paper proposes a model for predicting the dynamic motions of elastic robots made entirely of electro-active polymer gel. The requirement is that both the active and passive deformations be described simultaneously. Active deformations arise from a generated stress, while passive deformations are due to external forces, such as gravity and friction. We represent both in the same framework, considering them as the result of either electrochemical or mechanical interaction between the gel robot and the environment. We evaluated the model by comparing the results obtained through simulations and experiments using prototype gel robots. One experiment involved hitting an external object, and the other involved bending while resting upon a surface. We present an overview of the model, with a discussion of the experimental results and future work.

We present an algorithm for planning safe navigation strategies for biped robots moving in obstacle-cluttered environments. From a discrete set of plausible statically-stable, single-step motions, a forward dynamic programming approach is used to compute a sequence of feasible footstep locations. In contrast to existing navigation strategies for mobile robots, our method is a global method that takes into account the unique ability of legged robots such as bipedal humanoids to traverse obstacles by stepping over them. Heuristics designed to minimize the number and complexity of the step motions are used to encode cost functions used for searching a footstep transition graph. We show preliminary results of an experimental implementation of the algorithm using a model of the H6 humanoid navigating on an office floor littered with obstacles.

The development of a robot which has a flexible spine is presented. By embedding a multi-DOF (degree of freedom) soft structure into a robot body as a spine, the robot can increase its ability to absorb shock and the ability to work in various environments such as narrow places or human fields, and its motion could be more natural. The developed full-body human-form robot has a five-jointed flexible spine. Each joint (vertebra) has 3 DOFs. Between each two vertebrae is a "disk" made of silicone rubber. The spine is controlled by eight tendons, whose tensions can be controlled using tension-sensors and locally distributed microcontrollers. The paper describes the development of the flexible spine and the control of the posture of the spine and body.

This paper proposes a 3D depth flow generation method which measures 3D motion vector of every pixels between two time sequential images. First, the definition of 3D depth flow and a simple method in order to generate the 3D depth flow are described. Then the implementation of a real time 3D depth flow generation system using only a PC is presented, and experimental results are given. Finally, as an application, a walking human tracking task using mobile robot is included.

This paper describes the design and implementation of real-time depth map generation system with four key issues: 1) recursive (normalized) correlation technique; 2) cache optimization; 3) online consistency checking method; and 4) applying the MMX/SSE/sup R/ multimedia instructions set. The system is implemented on a standard PC/AT hardware with simple image capture board. Implementation details and system evaluations are also denoted. Furthermore, experiments with robots in real world are shown.

Introduces and describes a kinematic model based on a chemical reaction to control robots made of soft materials, poly (2-acrylamido-2-methylpropane sulfonic acid) gel (PAMPS gel). Experiments were conducted using a prototype mechanism whose energy is supplied by applying an electric field in a surfactant solution. We have verified the validity of our model by comparing the experimental and simulation results obtained by bending a flexible strip of PAMPS gel under a uniform electric field.

We propose a behavior learning method based on Bayesian networks and experience of interaction between human and robots, which does not need a priori knowledge and can be applied to human-robot interaction models. In this method, the behavior learning based on interaction experience was established. However, developers must adjust initial sensor state of the Bayesian network according to the user preference. In this paper, we propose a new method of state space construction for user adaptation based on introspection of interaction experience using genetic algorithms. We also give two examples: 1) obstacle avoidance tasks for mobile robots; and 2) symbol grounding for natural language instruction, for realization of user's adaptation of human-robot interaction.

Physically-based simulation software is commonly used for developing and testing low-level robot control algorithms. In order to facilitate the development and evaluation of higher-level robot behaviors, broader-based simulations are needed. Examples include software for simulating 3D vision, motion planning for obstacle avoidance, and integrating vision and planning. In addition to modeling the general interaction between the robot and its environment, the software can be used as a graphical user interface for directly controlling or interacting with a robot operating in the real world. This paper describes our current efforts toward building a large-scale software simulation framework for the development and testing of high-level behaviors for humanoid robots. We view this as a potential useful tool for the visualization and development of robotic systems, as well as an interactive, task-level programming interface for robots.

A map is a description which represents the structure of an environment. In order for a robot to move from a point to another efficiently in an environment, a map of the environment is the essential information. "Omni-View Sequence" is a kind of route representation which we have developed for robot navigation in a corridor environment. It contains a sequence of omnidirectional views as the information of a route. However it cannot be regarded as a map, because the start and the goal points of the route are both fixed. In this paper, we first propose a map named "view-sequenced map" which can represents a whole corridor environment on a floor in a building. Then we describe the method for acquiring a view-sequenced map automatically based on the exploration of a robot in a corridor using both stereo and omnidirectional vision. Finally experimental results of the map acquisition and the autonomous navigation are presented to show the feasibility of the proposed map representation.

Recently, view-based or appearance-based approaches have been attracting the interests of computer vision research. Based on a similar idea, we have proposed a view-based navigation method using a model of the route called the "view sequence." It contains a sequence of frontal views along a route memorized in the teaching run, and the recognition of the environment is realized based on the matching of the current view and memorized view sequence. In this paper, we discuss the required characteristics of the view for the view sequence, and evaluate our former method of generating views. Then we confirm that the stereo disparity satisfies the requirement of the view sequence through an experiment, and the disparity view sequence is applied for outdoor navigation. The experimental results indicate such views other than normal camera images can be utilized for our view-based navigation method.

A humanoid robot "H6" is developed as a platform for the research on perception-action coupling in intelligent behaviour of humanoid type robots. The H6 has the features as follows: 1) a body which has enough DOFs and each joint has enough torque for full body motion, 2) a PC/AT compatible on-board computer which is controlled by RT-linux so that low-level to high-level control is achieved simultaneously, 3) is self-contained and connected to a network via radio ethernet, 4) a 3D vision function can be applied. The H6 is expected to be a common test-bed for experiment and discussion for various aspects of intelligent humanoid robotics.

This paper describes a vision system which recognizes 3D objects in real-time by modeling the shapes of objects and matching the generated models. We develop the following methods for practically solving important problems of integration such as the estimation of sensor accuracy as well as real-time processing: 1) we reduce the computation of signed-distance, which is necessary to apply the marching cubes algorithm, and select the optimal resolution of models to be generated using an octree, thereby enabling us to generate hierarchical mesh models in real-time; 2) we apply spin-image matching by selecting the resolution of generated models and the coarse to-fine algorithm, consequently, we are able to efficiently match multiple objects of different sizes.

It is not simple for humanoid to acquire its actions automatically because: 1) it has very wide search space due to many DOF and is not fixed on the ground; 2) complex actions are difficult to describe using indirect evaluation functions; and 3) a humanoid is required to behave naturally like a human does. Kinematics adaptation generates an action outline which is a rough action description from teacher's action (motion capture data) by absorbing a difference of structures between the human and robot. Then a target action is acquired through dynamics adaptation by searching an action which is executable dynamically. Advantages of this framework are that: 1) the target action is described in an intuitive representation, 2) the search space is reduced in the neighborhood space, and 3) the acquired action will be a natural motion.

Describes what a friendly artifact is and the development of the first prototype robot system of a friendly artifact for multiple sensored interaction. The multiple sensored interaction is indispensable to realize the recognition of scene context, natural communication, self-expression and intuitive teaching which enables the robot to work besides humans in daily life. We developed a full-body prototype robot which works on a table. It has cameras, microphones, and a tactile sensor suit which covers its entire body. It also has a speaker to speak with, and has 15 DOF to realize gestures. A system with multi-threaded structure was built to realize multiple sensored interaction based on context and was implemented as a context based interaction system, the plausibility was confirmed in an experiment.

In this paper, a developmental construction methodology for a humanoid robot system is proposed. "Developmental" means that a versatile robot system is inherently a target to be developed incrementally, repetitively with inheriting results of former generations. Based on this methodology, the developmental system consists of (1) the hardware independent robot model for a skill development, (2) an action space representation for a functional development and (3) a robot body which has an onbody network for a physical development. On this system, five humanoid robots have been developed and locomotion behaviors have been realized.

View-based or appearance-based approaches have been attracting the interests of computer vision research. We have already proposed a visual view-based navigation method using a model of the route called the "view sequence", which contains a sequence of front views along a route memorized in the teaching run. In this paper, we apply an omnidirectional vision sensor to our view-based navigation and propose an extended model of a route called "omni-view sequence". The omnidirectional vision sensor is a desirable sensor for real-time view-based recognition of a mobile robot because all the information around the robot can be acquired simultaneously. The use of omnidirectional images contribute to a better navigation method in three ways: (1) it enables the robot to come back along the route using the same view sequence, (2) it improves the accuracy of the navigation and (3) it improves the robustness of the matching. These improved points are shown through experiments, and indoor navigation, including getting on and off an elevator, is demonstrated.

Simulation is an important issue in robotics research because it is essential for evaluating and predicting the behavior of a robot. However conventional simulators don't have the function of simulating the visual perception process. This indicates that it is impossible to simulate the vision-based behavior of a robot. In this paper, we propose a new simulation system "View Simulation System (VSS)", which is designed to simulate the behavior of a mobile robot equipped with vision. VSS has vision processing hardware, and can run any programs including vision processing for our real mobile robots. To show the feasibility of the simulator, we applied it to the evaluation of our visual navigation method.

The function that enables a robot to react when touched by a human is indispensable for a robot which enters our daily life. This paper describes the model and processing of full-body humanoid's tactile sensor suit for the human-robot interaction. The tactile sensor suit is first modeled to make a symbolized tactile data structure. Then conversion and compensation process is proposed to enable the robot to obtain accurate tactile data no matter what posture it takes. Then we explain the robot system and two experiments using a 36 DOF full-body humanoid: the first is the acquiring maps to process the tactile data, and second is the conformation to a human's shape as a human-robot contact interaction with proposed sensor suit model and processing.

A legged robot "JROB-1" is developed for a robotics research platform as a result of inter-university research program on intelligent robotics supported by the Ministry of Education Grant-in-Aid for Scientific Research on Priority Areas in Japan. The JROB-1 features: 1) self-contained, 2) RT-Linux running on PG/AT processes vision and sensor processing, motion planning and control, 3) connected to a network via radio Ethernet as to utilize networked resources, 4) Fujitsu color tracking vision board and Hitachi general purpose vision processing board, 5) all parts are commercially available, and 6) it is extensible with respect to sensor, sensor processing hardware and software. JROB-1 is expected to be a common testbed for experiment and intelligent robotics research by integrating perception and motion.

In this paper, a second generation remote-brained humanoid robot is presented which is developed for research on whole body action. Humanoid robots are important as a platform for the integration of techniques and algorithms acquired from research on manipulators, legged robots and so on. And they have new problems such as how to acquire, memorize, select and carry out various motions which use their whole bodies efficiently. In order to do research on these problems, a good robot body, which has whole body and enough performance for walking and getting up when it falls down is necessary. At the same time, a powerful brain is necessary which can be evolved through the body. As a solution to these demands, remote-brained approach was proposed and several humanoid robots was developed. Using these robots, several researches were done. For example, a brain framework called BeNet, an action acquisition using GA and NN and so on. They were done using a simple wireless connection. Interface between robot brain and its body to concentrate on a high-level problem. However this interface limited actions the robot can do simultaneously. In this paper, this old interface is taken to the next step. New interface has multiple actuator control methods which are switched on demand, and an onbody microprocessor network which controls actuators, measures sensors and interacts with a brain. Finally a new humanoid robot is developed on this interface.

In this paper, we propose an environmental description for robots to memorize temporality and spatiality of wide-range environment. The description is region-based and the environment is described as a mosaic of regions. The description consists of three components: panoramic labelled image that is a segmented color panoramic image, region database that has spatial and temporal information about each region, and regions relation network that describes adjacency state between regions. The description is constantly updated in the background of other visual processing. The description enables robots to detect changes in the environment, to grasp their characteristics, and to memorize them. Moreover, we implement the environmental description on the humanoid robot Saika and show how it expands the robot's behavior.

We propose an approach "online visual feature determination through discourse", which realizes the finding and the following task in a real environment. To segment the human's image from the complex background, it is possible to prepare many finds of basic visual features, and to combine them. The proposed approach is a solution for the problem of how to select and combine these visual features for each situation. Namely, a robot and a human make a conversation to search for the suitable visual features for the current situation. This idea enables the robot to find and follow a human successfully in a changeable background environment.

View changing because of vibration while walking is one of the most fundamental problem for a vision based legged robot. To overcome this difficulty, three key issues are denoted: a) integration of color segmentation, optical flow and stereo, which is able to apply to vibrating view using correlation hardware, b) software servo loop implemented as a kernel module for the purpose of soft actuation, and c) smooth walking pattern generation using solid model and dynamics simulator. Furthermore, a quadruped legged robot "JROB-1" is developed as a platform for the research on perception-action coupling in intelligent behavior of robots.

A robot body is not only an active unit but also a sensing unit. This paper describes a humanoid that integrates stereo vision, full-body tactile sensor, sound sensors and wrist force sensors into a sensor image. The sensor image reflects the current state of the environment and is regarded as all the input to the robot brain in this system. The brain program processes a series of sensor images and controls its body using them. The robot is remote-brained and designed to be a testbed to do research on sensor based behaviors of a full-body humanoid. This paper describes the design concept and the details of the system.

This paper discusses a stereo vision processing system which processes color and monochrome stereo video signals on a single vision processing board. Here, we propose a "field mixing" technique for multiplexing multiple video signals. A compact color stereo vision system based on this technique is developed for a mobile robot. This system can process multiple video signals simultaneously, and realizes flexible color stereo vision processing. In order to show the feasibility of this vision system, we installed it on our mobile robot, and implemented a correlation-based EZDF method for stereo tracking of an object. The experimental result of the tracking is shown.

We describe the method in which a visually guided swing motion for a 16 DOF two-armed bipedal robot is acquired by applying a GA (genetic algorithm) to a NN (neural network) controller. The evolutionary approach to the acquisition of various motions for robots has been successfully used by many researchers, but most studies have been carried out only through computer simulations. In this research, we adopt a real robot with a complicated body used in a noisy environment. The evolutionary processes are examined in. A virtual world constructed on a CRS-CS6400 parallel computer which simulates such factors as swing dynamics, visual processes noise reduction processes, and time lags in a control system. It took about and hours for an artificial evolution to create a successfully individual after 50 generations from an initial population of 200 unsuccessful genes. Using the NN decoded from the most successful individual of the last generation, a real two-armed bipedal robot that could swing successfully was obtained.

In this paper, we describe the design implementation of this real-time layer for remote-brained robot. "Remote-brained approach" is a paradigm for software research by separating the robot brain from the robot body. The interface from the body to the brain top-level software is also denoted. Furthermore, an application of the reactive motion of humanoid-type robot is described. The real-time facility is fundamentally important for a robot that behaves in the read-world. To accomplish both large scale intelligent software and real-time functionality, we design the system by dividing the brain into a low-level real-time layer and a high-level action decision layer. The action decision layer is put on a WS, and the real-time layer is put on a transputer network.

This article addresses the development of a light-weight, human-size and low-cost developing humanoid robot named Saika. Saika has a 2-DOF neck, two 5-DOF upper arms, a torso and a head. Several types of hands and forearms are developed. They are chosen depending upon the tasks to perform. The features of Saika are: (a) Saika as modularized to reduce the developing cost and to make maintenance easy, (b) the total weight of the head, the neck, the two upper arms and the torso is only eight kilograms and (c) most of the motors are installed inside the arms and the torso.

We present an approach to realizing a complex motion system by describing an asynchronous network of real time computational modules that represent dynamical systems with input, output and internal state completely. In our approach, the system is decomposed into parallel modules that calculate their assigned parts of the internal state at regular intervals. We stipulate the roles of the modules so that we can design each module independently and modify or extend the system easily. We have developed programming environments in which we can describe each module and the network briefly in programming languages and test it immediately in the real world. We describe how to design an understandable hierarchical network for desirable actions of an autonomous legged robot. The system has modules that run in parallel for generating motion patterns and actions. The described network was simulated in real time on a multiprocessor system and it generated desirable actions.

Previous work in vision-based mobile robotics have lacked models of the route which can be utilized for (1) localization, (2) steering angle determination, and (3) obstacle detection, simultaneously. In this paper, the authors propose a new visual representation of the route, the "view-sequenced route representation (VSRR)." The VSRR is a non-metrical model of the route, which contains a sequence of front view images along a route memorized in the recording run. In the autonomous run, the three types of recognition described above are achieved in real-time by matching between the current view image and the memorized view sequence using a correlation technique. the authors also developed an easy procedure for acquiring VSRRs, and a quick control procedure using VSRRs. VSRRs are especially useful for representing routes in corridors. Results of autonomous navigation using a two-wheeled robot in a real corridor are also presented.

This paper describes a method for tracking moving objects with multiple cooperative visual attention regions. Each attention region is tracked using a sum of absolute difference (SAD) correlation technique. The method is based on an implementation of a transputer-based multi-processor vision system with a correlation processor. The vision system provides a large number of attention regions, each of which can be tracked using fast SAD correlation performed in an LSI MEP. We call these regions windows since the region resembles a clipped section of a whole view. On the top of the system, we built a software manager which facilitates program image processing, motion control of a window, and cooperation control of windows. The cooperation is achieved using constraints, which are tree-structured, between windows. Cooperation of multiple attention regions enables one to track route predictable objects, deformable objects, and structured objects in a natural scene. As examples, passing car tracking, human arm tracking, and finger action recognition are shown.

We present design and implementation of a tactile sensor system, sensor suit, that covers the entire body of a robot. The sensor suit is designed to be soft and flexible and to have a large number of sensing regions. We have built the sensor suit using electrically conductive fabric and string. The current version of the sensor suit has 192 sensing regions. Each sensing region works as a binary switch in the current version. All of the signals from the sensor suit are gathered and superimposed on a visual image of the robot. The video multiplexer for the sensor signals is built on a field programmable gate array set. The construction of sensor suit system and its application with a full-body humanoid are presented.

We present a 35 DOF full-body humanoid is designed to be a testbed to integrate research tools for humanoid full-body behaviors. Each limb has 6 DOF. The neck has 3 DOF. Each hand has 4 DOF. The key idea of the system architecture is the remote-brained approach. This paper describes the design concept and the details of the system and introduces experiments to stand up using table for support and to perform action selection in reaching and grasping an object based on vision.

We present a vision-based real-time control system of a human-form robot in swing motion. The robot has 16 DOF joints and a camera which is designed to do research on human-form robot behaviors. The key idea to develop the vision based system for the robots are the remote-brained approach and real-time visual tracking. In this approach, the robot system is designed to have the brain and the body separate and to connect them by a radio link. The body sends the visual image to the brain and the body receives the motion, references. The basic algorithm to observe the swing motion by vision uses the optical flow generation. The control algorithm for flow-based visual feedback is described.

This paper consists of three parts. Part one introduces an ongoing inter-university research program on intelligent robotics. This program started in 1995 and will last for three years, supported by a grant from the Japanese Ministry of Education as a priority area for scientific research. The program is entitled "Research on Emergent Mechanisms for Machine Intelligence-A Tightly-Coupled Perception-Motion Behavior Approach." It aims to study mechanisms for the emergence of robotic intelligence and autonomy. Adopting a tightly coupled sensor-action approach, this program will tackle the theoretical foundations of machine intelligence, and experimentally study autonomous systems which perceive and behave in the real world. Part two discusses future directions for robotics, including issues, approaches and applications. Part three presents the author's view on a paradigm change for robotics research, and sketches out his group's R&D project, which aims at a break-through to blaze a trail for the robotics of the future.

Constructing an environment for robot software research, which can program and experiment in various robot behaviors, design of a software development platform becomes important problem. This paper describes a design of "Software Platform" for real-time large scale robot software, and its implementation in the "Remote-Brained Robot Project". The Software Platform is designed as three layers, "MOTHER, BRAIN, SENSOR-MOTOR". MOTHER consists of tools to produce and evolve BRAIN programs. For tools and libraries in such platform, there are two major problems. One is, there is tradeoff relationship between "Extension" and "Share". A way of overcoming this problem is described. The other is, the methodology is needed between low-level real-time parallel environment monitoring program and high-level software that takes much time. A system developed for remote-brained robots and its application are described. This system aims at developing many sorts of brain architecture and high level software that consists of various flexible multiprocess network.

We have focused on actions of a humanlike robot which has the whole body and realized actions such as "walk", "roll over" and "stand up". In this research, we design and implement the whole body humanlike robot that can perform a "carry objects" action. Keys to realizing a carrying action are (1) the robot system which has a lightweight whole body robot and a powerful brain which can process a multi joint model of the body, (2) the model environment which can generate walking patterns on the horizontal ground, stairs and a slope by moving its centroid and (3) gravity compensation based on modeling of the actuator.

We propose a computational model, BeNet, for describing the brains of autonomous robots and an approach to integrating functions of autonomous robots. BeNet is a network of asynchronously communicating processes that change their state and output vectors and their own frequencies determined by the designer. One can describe a modular system for a desirable behavior as a BeNet that interacts with the real world in real time. We design modules of a BeNet that calculate a part of the internal representation of the brain that is necessary for the desired behavior. The portions of internal representation such as action command, attention command, perceptual/sensational information are realized as messages among the modules. Building such a modular system helps you to develop a brain for a desirable behavior, modifying and extending the described system, since the information flow and the modules' roles are clear. We have implemented environments for developing BeNet on parallel computers. In these environments, you can describe BeNets that consist of processes of wide range in complexity and frequency of calculation, and test their behavior in the real world immediately. They are effective for building desirable behaviors of autonomous robots. They are demonstated for a ballboy robot, with subsystems for vision, motion and decision.

Presents a new type of robot which has two arms and two legs like an ape and is aimed at studying a variety of vision-based behaviors. The robot does not bring its own brain within the body. It leaves the brain in the mother environment and talks with it by radio links. The brain is raised in the mother environment inherited over generations. The key idea of the remote-brained approach is that of interfacing intelligent software systems with real robot bodies through wireless technology. In this framework the robot system can have a powerful vision system in the brain environment. The authors have applied this approach to formation of vision-based dynamic and intelligent behaviors of a multi-limbed mobile robot. In this paper the authors present an apelike robot with the remote-brained environment and describe vision-based experiments carried out with the apelike robot.

Presents a practical mobile robot system, called Hyper Scooter, which a human can ride on, and share access to the environment through visual information. The advantage of these capabilities is significant. A user can give his/her instructions to the robot without skillful programming operations. This is supported by the authors' high speed tracking vision system. To show the advantage, a complex task is presented as an example. After being instructed the way to accomplish the task by a human, Hyper Scooter does it without explicit environmental models.

Focusing attention on flexibility and intelligent reactivity in the real world, it is more important to build, not a robot that won't fall down, but a robot that can get up if it does fall down. This paper presents research on a two-armed bipedal robot, an apelike robot, which can perform biped walking, rolling over and standing up. The robot consists of a head, two arms, and two legs. The control system of the biped robot is designed based on the remote-brained approach in which a robot does not bring its own brain within the body and talks with it by radio links. This remote-brained approach enables a robot to have both a heavy brain with powerful computation and a lightweight body with multiple joints. The robot can keep balance while standing using tracking vision, detect whether it falls down or not by a set of vertical sensors, and perform a getting up motion by coordinating two arms and two legs. The developed system and experimental results are described with illustrated real examples.

We present a variety of vision-based adaptive and interactive behaviors in mechanical animals. The mechanical animal is a multi-legged robot designed as a remote-brained robots which does not bring its own brain within the body. It leaves the brain in the mother environment and talks with it by radio links. The brain is raised in the mother environment inherited over generations. The key idea of the remote-brained approach is that of interfacing intelligent software systems with real robot bodies through wireless technology. In this framework the robot system can have a powerful vision system in the brain environment. We have applied this approach toward formation of vision-based dynamic and intelligent behaviors of mechanical animals such as doglike robots and apelike robots. In this paper we introduce the remote-brained approach and describe some remote-brained robots and visual processes for adaptive and interactive behaviors with them.>

A novel task instruction method for future intelligent robots is presented, In our method, a robot learns reusable task plans by watching a human perform assembly tasks. Functional units and working algorithms for visual recognition and analysis of human action sequences are presented. The overall system is model based and integrated at the symbolic level. Temporal segmentation of a continuous task performance into meaningful units and identification of each operation is processed in real time by concurrent recognition processes under active attention control. Dependency among assembly operations in the recognized action sequence is analyzed, which results in a hierarchical task plan describing the higher level structure of the task. In another workspace with a different initial state, the system re-instantiates and executes the task plan to accomplish an equivalent goal. The effectiveness of our method is supported by experimental results with block assembly tasks.>

Graspless manipulation refers to operations in which a robot attempts to change the position and posture of an object without grasping it. Typical examples of graspless manipulation which have been studied so far are pushing and tumbling. This paper proposes a new method of graspless manipulation, pivoting, in which robot fingers maneuver an object in the same way that person moving furniture, as if an object walks on the floor by using appropriate vertices as virtual feet. This paper introduces the concept of this pivoting operation, describes its characteristics and analyzes it mathematically, points out the need for visual guidance for the precise performance of pivoting, and presents experiments with pivoting manipulation in connection with visual guidance.

A stereo vision system which consists of a stereo viewer and a processing system with viewing control mechanisms is presented. The stereo viewer is an adapter which, attached to a single camera, provides it with a stereo image by means of a set of four mirrors. Since the system can control the angle of mirrors, it can adjust them to operate during zooming, panning, and focusing in stereo viewing. The vision system uses a high-speed block matching processor to provide the essential functionality needed for motion tracking and stereo matching. With this system visual searching and stereo tracking can be implemented in real time. The authors analyze the optical structure of the stereo viewer, describe the vision and control system, and present some experimental examples of its use.

The authors describe a high-performance robot vision system that performs real-time tracking of moving objects, real-time optical flow computation, and high-speed depth map generation. The system was implemented as a transputer-based vision system augmented with a high-speed correlation processor. The transputer vision board was equipped with three image frame memories, each of which could be used simultaneously for image input, image processing, and image display. Thus, the system could devote all its computation power to image processing without waiting for image input or display. The vision board was also equipped with a standard image compression chip, used as a correlation processor. Using the chip, a very fast correlation-based robot vision system was developed. This system can also be used as a multiprocessor configuration to greatly improve performance.>

Functional units and working algorithms for real-time visual recognition of human pick-and-place action sequences are presented. The action recognizer consists of visual feature detectors, an action/environment model, and an attention stack. It generates a symbolic description of the observed action sequence. Given a different initial state, the system reinstantiates the recognized action sequence to carry out an equivalent assembly task. Experimental results on several assembly tasks support the effectiveness of the method.>

This paper describes a method to observe a moving object with multiple visual tracking agents. When a robot vision watches an object, it employs several tracking agents to monitor key features of the object. Each tracking agent is controlled by its visual feedback. In order to perform robust tracking, the authors introduce constraint control between agents. This method is effective in observing a motion of not only a single solid object but also a structured object such as a human hand. Several examples applied to 3D motion are shown.>