Robots are required to support people's work. In order to alleviate the burden on people, it is desirable that robot can automatically generate and execute complicated motions according to simple directions from people. However, there are multiple grasping methods for one object. In order to select a motion suitable for the direction, it is important to estimate candidates of grasping method for the object. In this research, we purpose to recall candidates of grasping position and hand shape from an object image. In learning, a network that outputs a plurality of grasping method candidates for one object image to each channel of a multi-channel image is used. At this time, a plurality of grasping methods are not learned at same time, learned one by one. The similar grasping method for the similar object shape is automatically clustered to each output channel in the learning process, and a grasping method having a characteristic difference is presented as a candidate. We show the usefulness of this method using experimental examples.

This paper proposes a system for an automatic detection of indoor scene events with interactive inquiry based on speech dialog and gesture recognition. The system detects the events that various objects are brought in or taken out by image recognition. The user of the system inquires the stored events in the past by pointing the objects or space and using speech dialog. Since automatic event detection may fail in complicated indoor scene, the system can use interactive inquiry to correct such failures.

Sign language is used for communicating to people with hearing difficulties. Recognition of a sign language image sequence is challenging because of the variety of hand shapes and hand motions. We propose a method to automatically construct a transitional structure(topology) of a Hidden Markov Model(HMM) for recognizing sign language words. Unlike conventional HMM, the constructed topology has branches and junctions in order to represent a flexible structure. The proposed method consists of segmentation of a motion, and construction of the topology from segments. The topology is constructed from an initial topology by modifying it. With experiments, we show the effectiveness of the proposed method.

This paper describes a method of probabilistic obstacle map building based on Bayesian estimation. Most active or passive obstacle sensors observe only the most frontal objects and any objects behind them are occluded. Since the observation of distant places includes large depth errors, a conventional method, which does not consider the sensor occlusion often, generate erroneous maps. We introduce a probabilistic observation model, which determines the visible objects. We first estimate probabilistic visibility from the current viewpoint by a Markov chain model based on the knowledge of the average sizes of obstacles and free areas. Then the likelihood of the observations based on the probabilistic visibility are estimated and then the posterior probability of each map grid are updated by Bayesian update rule. Experimental results show that more precise map building can be built by this method.

This paper describes a panoramic view-based navigation in outdoor environments. We have been developing a two-phase navigation method. In the training phase, the robot acquires image sequences along the desired route and automatically learns the route visually. In the subsequent autonomous navigation phase, the robot moves by localizing itself by comparing input images with the learned route representation. To be robust to changes of weather and seasons, an object-based comparison is adopted. Our previous method applied a support vector machine (SVM) algorithm to object recognition and localization and exhibited a satisfactory performance but was sometimes sensitive to the variation of the robot's heading. This paper thus extends the method to use panoramic images. By searching the image for the region which matches the model image the most, a new method can considerably improve the localization performance and provide the robot with globally correct directions to move

This paper describes a choice strategy to ease user's burdens for an interactive object recognition system when the system obtains multiple object candidates as a recognition result. First, we propose several methods to display the recognition result so as to make recognition of the candidate easy and hierarchical methods to reduce candidates per choice. We verify their effectiveness by subjective tests. Next, we propose a strategy to minimize time spent for choices. We divide the time into that for displayed candidates and that for speech dialog, and formulate each time to evaluate the strategy quantitatively. Last, we compare the strategy based on choice time with a subjective strategy

Personal service robots are expected to help people in their everyday life in the near future. Such robots must be able to not only move around but also perform various operations such as carrying a user-specified object or turning a TV on. Robots working in houses and offices have to deal with a vast variety of environments and operations. Since it is almost impossible to give the robots complete knowl edge in advance, on-site robot teaching will be important. We are developing a novel teaching framework called task model-based interactive teaching. A task model describes what knowledge is necessary for achieving a task. A robot examines the task model to determine missing pieces of knowledge, and asks the user to teach them. By leading the interaction with the user in this way, the user can teach important (focal) point easily and efficiently. This paper deals with a task of moving to a destination at a different floor; the task includes not only the movement but also the operation of recognizing and pushing elevator buttons. Experimental results show the feasibility of the proposed teaching framework.

This paper describes a view-based localization method using support vector machines in outdoor environments. We have been developing a two-phase vision-based navigation method. In the training phase, the robot acquires image sequences along the desired route and automatically learns the route visually. In the subsequent autonomous navigation phase, the robot moves by localizing itself based on the comparison between input images and the learned route representation. Our previous localization method uses an object recognition method which is robust to changes of weather and the seasons; however it has many parameters and threshold values to be manually adjusted. This paper, therefore, applies a support vector machine (SVM) algorithm to this object recognition problem. SVM is also applied to discriminating locations based on the recognition results. This two-stage SVM-based localization approach exhibits a satisfactory performance for real outdoor image data without any manual adjustment of parameters and threshold values.

This paper presents a knowledge-based person-centric human-robot interaction system using facial and hand gestures. In the proposed method, face detection and person identification are first made. With the knowledge of the known user face and hand poses are then classified from the later image frame by the subspace method and the gestures are finally recognized. The rules for interpreting the gestures are selected according to each specific user recognized by the facial image. The user's name and gesture commands are sent to the robot through a Software Platform for Agent and Knowledge Management (SPAK) to implement person-centric human-robot interaction. The effectiveness of this method has been demonstrated by interacting with a humanoid robot Robovie.

This paper describes an ego-motion estimation method by integrating multiple scan matching results. The method considers both the uncertainty of scan matching results and that of estimated ego-motions, and not only estimates the latest ego-motion but also updates previous ego-motions. The estimation process is formulated as an iterative one using Kalman filter. We implement the method by using an omnidirectional stereo-based scan matching method. Experimental results show the effectiveness of the proposed method.

This paper describes a calibration method of an omnidirectional stereo system. The system uses a pair of vertically-aligned catadioptric omnidirectional cameras, each of which is composed of a perspective camera and a hyperboloidal mirror, thus providing a single projection point. We divide the calibration into two steps. The first step estimates the image center and the aspect ratio by fitting an ellipse to the mirror boundary in the image. The second step estimates the focal length and the camera pose (position and orientation) including scale by using a calibration pattern and epipolar geometry. Experimental results show the effectiveness of the proposed calibration method.

This paper describes the navigation of a mobile robot in unknown static environments using an omnidirectional stereo and a laser range finder. The robot detects obstacles by the sensors, estimates the ego-motion, integrates the sensor data to generate a probabilistic occupancy map, and plans a safe motion. This paper focuses on the ego-motion estimation and the data integration for map generation. We extend our previous methods to increase the robustness of navigation. Experimental results show the feasibility of our navigation method.

This paper describes a mobile robot navigation method in dynamic environments. The method uses a real-time omnidirectional stereo, which can obtain panoramic range information of 360 degrees. From this panoramic range information, the robot estimates its ego-motion by comparing the current and the previous observations in order to integrate observations obtained at different positions. The uncertainty in the estimation is also calculated. Next, the robot recognizes and tracks moving obstacles. Finally, the robot plans a collision free path by a heuristic planner in space-time considering the velocity uncertainty of observed obstacles. Experimental results show the effectiveness of our method.

This paper describes a view-based outdoor navigation method. In the method, a user first guides a robot along a route. During this guided movement, the robot learns a sequence of images and a rough geometry of the route. The robot then moves autonomously along the route with localizing itself based on the comparison between the learned images and input images. Since appearances of objects in images may vary much according to changes of seasons and weather in outdoor scenes, a simple image comparison does not work. We, therefore, propose a comparison method in which the robot first recognizes objects in images using object models which allow for appearance variations, and then compares recognition results of learned and input images. We also developed a method which automatically selects key images used for the comparison from an image sequence. Successful autonomous navigation experiments in our campus under various conditions show the feasibility of the method.

This paper describes a method of parallel scheduling of planning and action to realize an efficient and reactive robotic system in dynamic environments. The method uses partial planning result, which comes from an iterative refinement planning, to determine feasible actions to be executed in parallel with the planning process. The method is applied to a multiple-camera multiple-person tracking problem. Simulation results show the effectiveness of the method.

This paper describes a map generation method using an omnidirectional stereo and a laser range finder. Omnidirectional stereo has an advantage of 3D range acquisition, while it may suffer from a low reliability and accuracy in range data. Laser range finders have advantage of reliable acquisition of data, while they usually obtain only 2D range information. By integrating these two sensors, a reliable map can be generated. Since the two sensors may detect different parts of an object, a separate probabilistic grid map is first generated by temporal integration of data from each sensor The resultant two maps are then integrated using a logical integration rule. An ego-motion estimation method is also described, which is necessary for integration of sensor data obtained at different positions. Experimental results on autonomous navigation in unknown environments show the feasibility of the method.

Proposes a novel concept of an intelligent navigator that can give a driver timely advice on safe and efficient driving. From both the current traffic conditions obtained from visual data and the driver's goals and preferences in driving, it autonomously generates advice and gives it to the driver. Not only can operational-level advice be generated, such as emergency braking due to an abrupt deceleration of the vehicle in front, but also tactical-level advice, such as lane changing due to a congested situation ahead. Two main components of the intelligent navigator - the advice generation system and the road scene recognition system - are explained. A three-level reasoning architecture is proposed for generating advice in dynamic and uncertain traffic environments. Online experiments using the prototype system show the potential feasibility of the proposed concept.

Proposes a method to schedule parallel execution of planning and action of a mobile robot. The method considers the following two types of parallelism: (1) acting while planning, if a partial planning result can be used to determine feasible actions, such actions can be executed while the planning process is still going; and (2) planning while acting, if the result of the current action is (at least partially) predictable, the planning for the next action can start in advance of the completion of the current action. The proposed method uses the notion of planning-action consistency to guide the scheduling. The method has been successfully applied to a mobile robot navigation problem under sensor uncertainty.

This paper proposes a method to autonomously extract stable visual landmarks from sensory data. Given a 2D occupancy map, a mobile robot first extracts vertical line features which are distinct and on vertical planar surfaces, because they are expected to be observed reliably from various viewpoints. Since the feature information such as the position and the length includes uncertainties due to errors of vision and motion of the robot, the robot then reduces the uncertainty by matching the planar surface containing the features to the map. As a result, the robot obtains modeled stable visual landmarks from the extracted features. These processes are performed online in order to adapt to actual changes of lighting and scene depending on the robot's view. Experimental results in various scenes show the validity of the proposed method.

This paper describes a real-time omnidirectional stereo system and its application to obstacle detection and tracking for a mobile robot. The stereo system uses two omnidirectional cameras aligned vertically. The images from the cameras are converted into panoramic images, which are then examined for stereo matching along vertical epipolar lines. A PC cluster system composed of 6 PCs can generate omnidirectional range data of 720/spl times/100 pixels with disparity range of 80 (about 5 frames per second). For obstacle detection, a map of static obstacles is first generated. The candidates for moving obstacles are then extracted by comparing the current observation with the map. The temporal correspondence between the candidates are established based on their estimated position and velocity which are calculated using Kalman filter-based tracking. Experimental results for a real scene are described.

With the increase in the number of senior citizens, there is a growing demand for human-friendly wheelchairs as mobility aids. One of the main issues in the robotic wheelchair research is autonomous obstacle avoidance for safety. However, this is difficult because most moving obstacles in the real world are human beings. They sometimes change their motion abruptly. The paper presents an intelligent wheelchair that can avoid collisions with such human pedestrians safely and comfortably for each other. We assume that the information whether or not a pedestrian has noticed the wheelchair and which direction he/she wants to go can appear in the face direction. Thus our intelligent wheelchair is continuously observing the pedestrian's face in its front area, realizing smooth passing by changing its collision avoidance strategy based on the face information and the range data from the ultrasonic sensors. Experimental results show the effectiveness and comfortableness of the proposed method.

Proposes a probabilistic method for detecting people in a corridor using a sequence of color images. In a learning phase, the templates of people and the color properties of skins, a background without shadows, and a background with shadows of people are obtained. In a detection phase, the probabilities of the color of each pixel belonging to various objects are computed. Then templates of a person are placed at various image positions and the criterion of matching to a person is computed for each position. If a template is matched to a person, a more precise position of the person is determined near the matched position. We carry out an experiment using real image sequences to show the effectiveness of the method.

This paper describes a method of tracking a person with 3D translation and rotation by integrating optical flow and depth. The target region is first extracted based on the probability of each pixel belonging to the target person. The target state (3D position, posture, motion) is estimated based on the shape and the position of the target region in addition to optical flow and depth. Multiple target states are maintained when the image measurements give rise to ambiguities about the target state. Experimental results with real image sequences show the effectiveness of our method.

We presented an intelligent wheelchair whose motion can be controlled by the user's face direction. We propose to add intelligence to our wheelchair when the user is not riding. It can recognize the user's face and can move according to the gestures made by the user. Gesture is a good means to give commands because it can be used in noisy conditions and when we would not like to speak loudly in public places. However, environments where wheelchairs are used cannot be controlled. This makes gestures recognition difficult. We propose an interactive way to solve this problem. When the wheelchair is not certain about the meaning of the user's gesture, it guesses the meaning and moves a little accordingly to show its guess to the user. Then it observes the user's response, judging whether its guess is correct or not. This guess-action-observation cycle is repeated until the wheelchair can understand the user's intention.

In soccer games, understanding the movement of players and the ball is essential for the analysis of matches or tactics. In this paper, we present a system to track the players and ball and to estimate their positions from video images. Our system tracks the players by extracting their shirt and pants regions and can cope with the posture change and occlusion by considering their colors, positions, and velocities in the image. The system extracts ball candidates by using the color and motion information, and determines the ball among them based on motion continuity. To determine the player who is holding the ball, the position of players on the field and the 3D position of the ball are estimated. The ball position is estimated by fitting a physical model of movement in 3D space to the observed ball trajectory. Experimental results on real image sequences show the effectiveness of the system.

This paper proposes a method for the rapid and precise estimation of human hand postures by combining 2D appearance matching and 3D model-based fitting. First a rough posture estimate is obtained by image indexing. Each possible hand appearance generated from a given 3D shape model is labeled by an index obtained by PCA compression and registered with its 3D model parameters in advance. By retrieving the index of the input image, the method can obtain the matched appearance image and its 3D parameters rapidly. Then, starting from the obtained rough estimate, it estimates the posture and moreover refines the given initial 3D model by model-fitting. The sequential activation of the two processes in every frame gives the precise posture estimate rapidly. The effectiveness of the method is shown by experimental results.

Describes a method of modeling the motion uncertainty of moving obstacles and its application to mobile robot motion planning. The method explicitly considers three sources of motion uncertainty: path ambiguity, velocity uncertainty, and observation uncertainty. The model is represented by a probabilistic distribution over possible position an a path of a moving obstacle. Using this model, the best robot motion is selected which minimizes the expected time of reaching the destination. By considering not the range but the distribution of the uncertainty, more efficient behaviors of the robot are realized.

Humans can communicate with each other by gestures. This paper presents a robot system with such capability. The robot detects its user by recognizing his/her face. It can accept his/her commands given by gestures. The user may use unknown gestures to the robot. If the robot does not respond to his/her gesture, the user usually iterates the same gesture. The robot detects this repetitive pattern as an intentional gesture by which the user wants to give it some order. Then it shows a little action according to the meaning of the gesture that the robot guesses. It observes the user's reaction to its action. If he/she continues the same pattern gesture, the robot considers that its understanding is right, completing the action. It also registers the pattern as a gesture of the guessed meaning. Otherwise, it iterates the same procedure by taking another action as a candidate of the meaning. We have implemented such interactive capability on our intelligent wheelchair.

With the increase in the number of senior citizens, there is a growing demand for human-friendly wheelchairs as mobility aids. One of the main issues in robotic wheelchair research is autonomous obstacle avoidance for safety. However, this is difficult because most of the moving obstacles in the real world are human beings, and they sometimes change their motion abruptly. This paper presents an intelligent wheelchair that can avoid collisions with such pedestrians (and each other) safely and comfortably. We assume that the information as to whether or not a pedestrian has noticed the wheelchair and which direction he/she wants to go can appear in the direction of the pedestrian's face. Thus, our intelligent wheelchair continuously observes the pedestrian's face in its frontal area, realizing smooth passing by changing its collision-avoidance strategy based on the face information and the range data from ultrasonic sensors. The effectiveness of the proposed method is shown by a comparative experiment with a method using only ultrasonic sensors.

Human interfaces are usually designed to respond only to intentional human behaviors. However, humans show unintentional behaviors as well. They can convey useful information to realize user-friendly human interfaces. This paper presents how to combine observations of both types of behaviors by taking two human- machine systems: a gesture-based interface and an intelligent wheelchair. In the first system, intentional hand gestures are chosen using unintentional behaviors. In the second system, near unintentional behaviors following intentional behaviors can be used to control the wheelchair motion. Experimental systems working in real time have been developed. Operational experiments prove our approach promising.

With the increase of senior citizens, there is a growing demand for human-friendly wheelchairs as mobility aids. The paper proposes the concept of a robotic wheelchair to meet this need. It has three motion control modes. One is an autonomous control mode. The wheelchair has autonomous navigation capabilities, such as avoiding obstacles and moving straight toward a target object or along a corridor using ultrasonic and vision sensors. Another is a manual control mode with minimum intentional operations. The wheelchair observes the user's face by vision, moving in the direction indicated by the use's face movements. The third is a remote control mode. When the user is not riding, it can find him/her using face recognition. The user can make it come or go by hand gestures. We have developed an experimental robotic wheelchair based on the above concept. Experimental results prove our approach promising.

Proposes a method to autonomously select stable visual landmarks from observed features by stereo vision and a given 2D obstacle map. The robot selects as stable landmarks vertical line segments which are distinct and on a vertical plane, because they are expected to be observed reliably from various viewpoints. Due to the vision and motion error of the robot, the observed feature positions include uncertainty. This uncertainty can be reduced by matching the detected vertical plane which includes the features to a known plane in the map. The position of a selected feature is modeled by a probabilistic distribution on the known plane. The selection and modeling process is performed online to adapt to an actual lighting and background condition which varies depending on viewpoints. When the robot moves, it uses several, less uncertain landmarks to estimate its motion. Experimental results in real scenes show the validity of the proposed method.

With the increase in the number of senior citizens, there is a growing demand for human-friendly wheelchairs as mobility aids. To meet this need, we proposed a robotic wheelchair which can be controlled by turning our face in the direction where we would like to go. Although it can be used easily, there is a problem that unintentional movements of our face may interfere with the wheelchair motion. The paper presents our new version of the wheelchair improved by observing both user and environment. It effectively integrates autonomous capabilities and the interface by face direction. It uses the sensor information obtained for autonomous navigation to solve the problem with the control by face direction. Also, if it can understand the user's intentions from observing the face, it chooses an appropriate autonomous navigation function to reduce the user's burden of operation.

We need a reliable visual surveillance system that can work under various disturbances, such as changes of lighting and movements of background objects. We propose a method exploiting information over space and time. First, optical flow is computed, and then regions with uniform flow are extracted. The regions are hypothesized as parts of detection targets. Assuming that the targets move in constant directions at constant speeds, we predict a path of each region and vote it in the space-time cube. If enough number of votes support a particular path, the method reports the detection of the intruder. We have implemented this method using multiple DSP boards and constructed a system that works in real time. Experimental results show that the system can detect intruders walking with many objects moving in their backgrounds.

This paper describes a method of extracting a moving object in a 3D space in a cluttered background by integrating optical flow and edges. The method extracts objects even if their appearance change and the flow vectors of the objects are not uniform. We segment the image into regions so that the motion of each region may be approximated by the 3D motion of a plane. In order to represent the whole contour of the regions even in a cluttered background, the brightness edges near the motion boundary are accumulated over frames. The contour of the region is composed of the motion boundary and the accumulated edges. The object is composed of adjacent regions which have similar 3D motion parameters. Experimental results for synthetic and real images show that this method extracts objects even if the appearance of the objects change drastically.

Proposes a method of removing ambiguities in robot tasks by a multimodal human-robot interface consisting of verbal and nonverbal communication. Such ambiguities often arise from failures of the robot vision system. However, it is not easy to solve this problem only by improving computer vision techniques. Thus, our robot asks a human such a question that a natural reply to it will contain helpful information to adapt the vision system for the current situation. We present a robot system that can bring the object ordered by a human by verbal and nonverbal behaviors.

This paper describes a method to track a person by integrating two cues, optical flow and uniform brightness regions, where optical flow cannot be obtained. This method works even if tracking with either optical flow or uniform brightness regions may fail. The proposed method has been implemented on a real time image processor with multiple DSPs and successfully tracked a target person using real image sequences.

With the increase of senior citizens, there is a growing demand for human-friendly wheelchairs as mobility aids. The paper proposes a concept of an intelligent wheelchair to meet this need. It can understand human intentions by observing the user's nonverbal behaviors and can move in accordance with the user's wish with minimum human operations. The paper also describes our experimental robotic wheelchair system. Human intentions appear most on the face. Thus, the experimental system observes the human face, computing its direction. As the first step toward the intelligent wheelchair, we have made experiments on controlling the system's motion by the face direction. Experimental results prove our approach promising.

This paper proposes a novel method of scheduling parallel execution of planning and action for a vision-motion planning problem. A planning process can be viewed as a process of gradually reducing the plan candidates towards the final commitment to one plan. Using criteria on deciding if an action sequence is consistent with the remaining plan candidates (consistency criterion), and on when to commit to the final plan (commitment criterion), an appropriate action sequence is selected and executed while the planning process is still continuing. Preliminary experimental results including the comparison with a sequential method show that the proposed method is promising.

This paper proposes a method of removing ambiguities in robot tasks by a multimodal human-robot interface consisting of verbal and nonverbal communication. Such ambiguities often arise from failures of robot vision systems. However, it is not easy to solve this problem only by improving computer vision techniques. Thus, our robot asks a human such a question that a natural reply to it will contain helpful information to adapt the vision system to the current situation. We present a robot system that can bring objects ordered by a human through verbal and nonverbal behaviors.

Describes a method of planning of observation and motion for a mobile robot to interpret a road intersection scene and to reach the intersection as soon as possible considering the uncertainty of interpretation. From a monocular color image, candidate regions are extracted for objects which are related to the intersection type. The probabilities of the regions coming from the objects are calculated from probabilistic models of the objects. From the probabilities of the regions and the relation between the intersection type and the objects, the current probability of intersection types is calculated. If the intersection type is ambiguous, the robot plans the observation and motion which minimize the expectation of the cost (time) to complete the task. The robot takes these actions and tries to determine the intersection type again. This process is iterated until the task is completed. The experimental result is shown for an actual intersection scene.

Presently, mobile robots are navigated by means of a number of methods, using navigating systems such as the sonar-sensing system or the visual-sensing system. These systems each have their strengths and weaknesses. To fully utilise the strengths of both the sonar and visual sensing systems, this paper proposes a fusion of navigating methods involving both the sonar and visual systems as primary sources to produce a fast, efficient and reliable obstacle-avoiding and navigating system. Furthermore, to further enhance a better perception of the surroundings and to improve the navigation capabilities of the mobile robot, active sensing modules are also included. The result is an active sensor fusion system for the collision avoiding behaviour of mobile robots, This behaviour can then be incorporated into other purposive behaviours (e.g. object seeking, path finding etc.). The validity of this system is also shown in real robot experiments.

The parametric eigenspace method was proposed by Murase-Nayar (1995) to recognize objects and their poses. It could be applied to robot navigation to locate the robot position. However, since similar images may often be taken at multiple locations in real scenes, it cannot always give the robot position reliably with a single image input. This problem can be solved using active vision, that is, combining localization results for images taken at multiple camera positions. Since similar images are projected to points close to one another in the eigenspace, we can tell before actual navigation when we cannot expect reliable localization results with a single image by examining the eigenspace. Moreover, further analysis of the eigenspace can give the best action sequences of camera motion to efficiently localize the robot position. This paper presents such an eigenspace analysis method. Experimental results show the effectiveness of the method.

This paper deals with planning of visual navigation strategies for a mobile robot under uncertainty of both visual data and dead reckoning data. When a robot passes through a narrow space, it has to move slowly while carefully observing surrounding objects and estimating the distances to the objects precisely. On the other hand, the robot can move fast in a widely open space. A novel method is proposed which can adaptively determine the speed of the robot and the visual landmarks, thereby realizing an efficient navigation of the robot. Experimental results show the feasibility of the method.

We need a reliable moving object detection method that can work under various disturbances, such as changes of lighting and movements of background objects, to realize practical automatic surveillance systems. Using local information alone could not achieve this goal. Hence, we propose a method exploiting information over space and time. Optical flow is first computed, and then regions with uniform flow are extracted. The regions are hypothesized as parts of detection targets. Assuming that the targets pass in the scene at constant speeds, we predict the path of each region, voting it in the space-time cube as in the Hough transform. If enough number of votes support a particular path, the method reports the detection of a moving object. Experimental results show that it can detect walking people in complex scenes with moving background objects, such as trees shaking in the wind.

This paper describes a method of determining contours of moving objects in a cluttered scene by integrating optical flow and edges. If the motion of an object is similar to that of the background, the contour is not determined only by optical flow. If the background of a scene is cluttered, the contour is not determined only from edges because many edges may be extracted in the background and no edges may be extracted on some parts of the contour. In the proposed method, the contour is determined by using optical flow and edges in a long sequence. The whole contour of a moving object is eventually obtained by accumulating edges near motion boundaries over an image sequence. The method can also determine the occlusion relation of two overlapping objects by checking if edges exist on the predicted contours of objects. Experimental results for synthetic and real images show the usefulness of the method.

This paper describes a method of recognition of intersection scenes by attentive observation considering the uncertainty of recognition, for mobile robots on roads. From a monocular color image, homogeneous color regions are extracted. Probabilities of the regions coming from specific objects are calculated. From these probabilities and the relationship between these objects and intersection types, the current probability distribution of intersection types is calculated. If the entropy of the probability distribution is lower than a certain threshold, the robot adopts the best hypothesis. Otherwise, the robot selects and observes the part which can minimize the expectation of the entropy attentively. These actions are iterated until the entropy becomes lower than the threshold. The experimental results are shown for actual intersection scenes including a white mark, a curve mirror and so on.

Geometric invariants from multiple views provide useful information for 3D object recognition. However, conventional object recognition methods using invariants based on point features cannot achieve efficient recognition because of large amount of combinations of point features in invariant calculation. To avoid this problem, the authors propose to use more complex features. They adopt arrow junctions and conics as complex features because man-made objects have often trihedral polyhedra (eg. parallelepiped) and circles and they make arrow junctions and conics in images, respectively. The multiple view affine invariance theory can be directly used for arrow junctions. For conics, they propose two types of invariants. They have developed an object recognition method exploiting these invariants. In addition to the recognition method with two input images, they propose a recognition method that needs only a single input image by substituting an image of a target object stored in the model library. Experimental results using 240 pair of images for 24 objects confirm the usefulness of the methods.

This paper presents a method for spatiotemporal segmentation of long image sequences of scenes which include multiple independently moving objects, based on the minimum description length (MDL) principle. First, a family of motion models is constructed, each of which corresponds to a physically meaningful motion such as translation with constant velocity or a combination of translation and rotation. Then, the motion description length is formulated. When an object changes the type of the motion or a new part of an object appears, the corresponding temporal or spatial segmentation is carried out. Ambiguous segmentation of two consecutive images can be resolved by minimizing the motion description length in a long sequence of images. Experiments on several real image sequences show the validity of our method.

This paper describes real time object tracking by extracting optical flows from a sequence of images. Optical flows are calculated based on the generalized gradient model. Our method detects a moving object, and tracks it from the optical flow data. Assuming that pixels corresponding to the same object have similar flow vectors, we extract a connected region with similar flow vectors. We apply this method to multiple object tracking. When two objects overlap, foreground object is recognized, and each object is tracked without confusion. Optical flow extraction and object tracking are executed in realtime by a special image processor.

This paper presents a human-robot interface system that enables a user to move a robot by moving his hand. The system adopts uncalibrated stereo vision based on the affine invariants from multiple views. The system can interpret hand gestures both in the user-centered frame and in the world-fixed frame. Suppose the user wants to indicate the direction of a mobile robot motion by the direction of his hand. If he moves his hand forward, the robot moves straight ahead regardless of his body position, that is, regardless of the hand direction in the world by the former interpretation. By the latter interpretation, however, the robot turns towards the direction indicated by the hand. The former is useful when we operate a teleoperation robot while watching the images sent from the robot. Operation experiments show the usefulness of the system.

Motion-based image segmentation becomes inherently ambiguous when apparent motions of different objects are locally or globally similar during a period. To disambiguate the segmentation, temporal coherence between the local image motion at each edge point and the apparent motion of every object is examined over a long sequence. The point is grouped into that segment of the object whose apparent motion is temporally most coherent with the local image motion at the point.>

This paper presents a method for spatiotemporal segmentation of long sequences of images which include multiple independently moving objects, based on the Minimum Description Length (MDL) principle. Spatiotemporal (ST) segments in the image sequence are extracted, each of which consists of edge segments having similar motions. First, we construct a family of motion models, each of which is completely determined by its specified set of equations. Then we formulate the motion description length in a long sequence based on these sets of equations. The motion state of an object at a given moment is determined by finding the model with shortest description length. Temporal segmentation is carried out when the motion state is found to have changed. At the same time, the spatial segmentation is globally optimized in such a way that the motion description of the entire scene reaches a minimum.>

This paper describes a new method of modeling an environment in terms of obstacles and free spaces from a set of 3D segments obtained by stereo vision. Since live stereo vision provides only the position of segments, it is necessary to determine whether a region formed by the segments is an obstacle or a free space. The ambiguities and the uncertainties in the obtained data must be considered in modeling. The final output of the proposed method is a set of possible situations of the environment and their probabilities; each situation consists of the description of obstacles and critical regions between the obstacles. Experimental results for a real scene are described.>

This paper presents a vision-based human-computer interface system that enables the user to move a target object in 3D CG world by moving his/her hand. Unlike conventional systems, the system does not need any camera calibration. In addition, it uses a user-centered frame so that the user can control the object by hand motions matched with human intuition. For example, the user can move the object forward by moving his/her hand forward even if he/she has changed his/her body position. The system is based on the multiple view affine invariance theory. It calculates hand positions as invariant coordinates in the basis derived from four points on the user's body. Actual system implementation and experimental results show the usefulness of the system.>

This paper describes a navigation method for a mobile robot by attentive observation. A 3D scene description is constructed with the constrained Delaunay triangulation and a path to a given destination is determined from the description. If any reliable path is not found, the unknown image region on the most promising path is observed attentively. The newly obtained data is integrated to the original description and a path is searched for in the new description. The experimental results are shown.>

An optical flow determination method is proposed which derives at a location more than two gradient constraint equations with a set of orientation-selective spatial Gaussian filters. The uncertainty measure of an optical flow is also obtained from the equations.>

A method to obtain the optimal motion description from two consecutive images including multiple moving parts is presented. It copes with segmentation and motion estimation problems. Segmentation is necessary for motion estimation of each part, and vice versa. The authors propose to use an information measure approach, based on comparisons between an individual (or pixel) and a class (or set of pixels). First, the motion of an edge segment is optimally modeled. Next, merging and splitting processes are iterated until the minimum description is obtained for the whole image. As a result, the image is segmented into several regions, each of which is represented by an edge segment list, and, at the same time, the maximum likelihood motion estimation is obtained for each region. Experiments performed on real images are shown.>

A qualitative approach to quantitatively recovering the shape and pose of a straight homogeneous generalized cylinder (SHGC) based on a weak Lambertian assumption is described. This assumption relaxes the strict cosine law of the Lambertian reflection model. The method does not need to know the lighting condition or surface albedo. The image of the projection of the axis of an SHGC is extracted. The slant angle of the SHGC is estimated using the weak Lambertian assumption along an extremal cross-section curve. The location of the SHGC's 3-D axis on other parallels is located. As a result, the pose (slant and tilt) and the shape (the shape of the cross section, the location of the axis, and the sweeping function) of an SHGC are recovered. Experimental results for both synthesized and real images are shown.>

A central problem in estimating multiple image motions is that 2-D motion estimation and region segmentation are mutually dependent. The authors present a new region description method for dealing with this mutual dependence problem. Segmentation and motion estimation are simultaneously performed by a clustering process based on color, motion, and pixel position. As a result of the clustering, an image is decomposed into region fragments. Each fragment is characterized by distribution parameters of color, pixel positions, and spatiotemporal intensity gradients. The image is described by the parameters of the region fragments; 2-D motion vectors for each fragment are obtained from the distribution parameters of the intensity gradients. By combining those features in the clustering process, regions are segmented more precisely, motion boundaries are not blurred, and the 2-D motions are obtained even in noisy areas. Experimental results are presented.>

High variability of object features and bad local feature separation of object classes are the main causes for the difficulties encountered during the interpretation of ground-level natural scenes. The authors propose a knowledge-based scene classification and segmentation method. The first stage carries out object classification on the basis of known local feature default models. It then searches for regions which satisfy qualitative constraints in the classified object images and selects good, approximate object region candidates, using multiple features. This approach is constrained in a top-down manner by the local feature default model and qualitative constraints while at the same time operating in a data-driven style through automatic threshold value and window size selection mechanisms. The second stage extends and refines the result of the first stage by performing a constrained search for object boundaries.>

This paper describes an approach to automatic landmark selection for navigation of a vision guided vehicle. The system works in two phases. In the first phase, the vehicle is driven by a human through a corridor. During this movement, a vision system extracts features such as doors or staircases. The category and the location of extracted features are recorded in a map. In the second phase, given a destination by a human, the system examines the map and selects a sequence of landmarks which guide the vehicle to the destination in the minimum time. Both the uncertainty of the motion of dead reckoning and that of visual data are considered in landmark selection. Preliminary experiments are performed for a autonomous vehicle which moves in the authors' building.

Weak Lambertian assumption is proposed and used to determine shape and pose of cylindrical objects from a monocular intensity image. The method does not require the knowledge of lighting conditions (light intensity and lighting direction), surface properties, or albedos. Experimental results for both synthesized and real images showing the validity of the method are presented.>

A framework for bottom-up initial segmentation of color images is proposed in which the role of initial segmentation is restricted to generating indices into a knowledge base of object models. The importance of primitive knowledge, feature integration and feature salience is discussed in the case of color and texture.>

In order to overcome the shortcoming of the current stereo vision, a reliable stereo method is proposed making use of the reliability of features in determining correspondence. A rich description of a scene is obtained by integrating range data and color data. Interpretation of a scene requires the use of the knowledge or models of objects. Based on the knowledge, most effective process is triggered at each stage depending on what kind of information is obtained at that stage. In this paradigm, a monocular image of a 3D scene can also be interpreted. The author describes applications of 3D computer vision and his approach to a more flexible 3D computer vision.>

The authors describe a framework for planning of vision and motion for a mobile robot. For planning in a real world, the uncertainty and the cost of visual recognition are important issues. A robot has to consider a tradeoff between the cost of visual recognition and the effect of information obtained by recognition. A problem is to generate a sequence of vision and motion operations based on sensor information which is an integration of the current information and the predicted next sensor data. The problem is solved by recursive prediction of sensor information and the recursive search of operations. As an example of sensor modeling, a model of stereo vision is described in which correspondence of wrong pairs of features as well as quantization error are considered. Using the framework, a robot can successfully generate a plan for real-world problem.>

The authors describe a reliable feature-based stereo matching method which determines more reliable pairs of matching edges earlier than less reliable ones and makes use of previous matching results. The contrast of an edge point is used as the measure of reliability of the match because an edge with high contrast has high positional directional accuracy. The decisions on whether there are correspondences between pairs of lower contrast edges or not become easier because previously found more reliable matching results are available to reduce the occurrences of ambiguous matches. Experiments with complicated indoor scenes proved that the proposed method is better than those which do not consider reliability.>

In this paper, we present a method which interprets a monocular color image using three dimensional( 3-D) information extracted dim ing the interpretation process. First, an inpiit im- age is segmented into regions each of which has uniform brightness and color. Then the regions are interpreted utilizing knowledge about objects such as color, size and shape, and candidate re- gions for the objects are determined. For some candidate regions, the 3-D information can be ex- tracted such as position, direction, or size. Once the 3-D information is obtained, it can improve the current result of interpretation and constrains the interpretation for other regions. By iterating this process, the current interpretation result is improved and more accurate one is obtained. As the final result, the interpretation labels for the re- gions in the image and 3-D relationships between the objects are obtained. Experiment for several road scenes has proved the validity of the method.

A new framework of a hierarchical vision-motion planning for a mobile robot under uncertainty is proposed. An optimal plan is generated by two-level planning: global route selection and local path planning. In global route selection, a sequence of observation points to acquire sufficient information to reach a destination is determined. Both the cost and the uncertainty of vision are considered in this planning. In local path planning, given two successive observation points, trajectories, moving speeds, and reference points for robot localization are determined so that a robot can reach the second point safely with a minimum cost. Both the error in localization and that in motion control are considered in this planning. A local path planner is repeatedly invoked in global route selection to determine an actual path between observation points. Our hierarchical planner can generate an optimal plan for a mobile robot planning problem. I. Introduction A mobile robot with vision moving to ...

A qualitative approach to quantita- tive recovery of cylindrical shape, pose and illumi- nant cpditiop is described. ''Weask Lambertian gs- sumption is introduced as a qualitative constraint on shading, which can relax the requirements for shape from shading methods in two ways: one is that it does not strictly constrain the property of the perfectly Lambertian surface of objects, and the other is that object surfaces may include spec- ular component in addition to diffused one. The method does not need to know the lighting con- dition (light intensity and lighting direction) or surface albedos. Input scenes include cylindrical objects each of which has a cylindrical surface and planar one as its cross-section. First, an actual lighting condition is transformed into a normal- ized lighting one, and an equation which relates the cross-section contour on the image plane to the shape parameter of the cylindrical object is derived based on the Weak Lambertian Assumption. In the case of scenes including plural cylindrical ob- jects, we can estimate the actual lighting direction and albedos of both diffused and specular com- ponents for each surface. Moreover, we can infer relative configuration of plural objects by making a comparison between an input image and a recon- structed image synthesized by using the estimated actual lighting direction and albedos. Experimen- tal results for both synthesized and real images are shown.

We introduce utilizations of T- junctions for stereo matching and detecition of T-junctions. By identifying T-junctions we can improve matchings near occluding con4ours. T- junction is also useful for a horizontal line to de- termtine whether or not the disparity can be inter- polated because the interpolation is allowed only if none of the terminals of the horizontal line relate with T-junctions. Edges are extracted as zero-crossings of a Lapla- cian Gaussian filtered image. We analyze the re- sponse of a Laplacian of a Gaussian filtered image at ideal T-junctions and obtain a method to de- tected T-junctions from a monocular image:.

We propose a method for model- based scene segmentation which separates object instances from the background in the scene. An object model is composed of parameterized geo- metric primitives constrained to each other accord- ing to its shape and size variations, and the rela- tionships to other object classes. As sensory data, we use a pair of color stereo images which provides color and disparity information. Using these infor- mation and the constraints in model descriptions, we approach to the problems of scene segmenta- tion and object recognition. First, we construct a height map, which shows a top view of the input scene, transforming the disparity information into 3-D space based on the height and tilt of the camera system. Using the size constraints of the object model, we separate candidate areas for object instances from the back- ground. Next, parts of the model are extracted from the scene descriptions inside each candidate area, and free parameters in the object model are determined so that the instance can be best fitted to the model. Finally, uncertainty measure for pla- nar patches is defined and applied to the extracted parts. We show the preliminary results using car models.

An approach to geometrical reasoning for world model representations that are based on planar surfaces from range and video images is described. The geometrical reasoning is regarded as an inferring process of the spatial extent of primal surfaces derived from the range image at an early stage. The inferring process has two subprocesses: expansion of a primal surface using a directional uncertainty defined by the moments around the axes on the plane fitted to it, and the determination of the boundary shape of the expanded surface using constraints on the spatial relationships between the observed data. Experimental results that were applied to road scenes in which the inferring process proved useful for the integration process of video and range image sequences are discussed.>

The authors propose a framework of unified planning of vision and motion with uncertainty. They use a probabilistic model to represent uncertainty and use statistical decision theory to make a unified plan of vision and motion. They describe a method of predicting the information acquired by a sensing operation and formulate the planning problem in a recurrence formula. They analyze vision-motion planning problem in a simple example and conclude that the combination of dynamic programming and hill-climbing is useful. Simulation results show the validity of the approach.>

Describes two topics: (1) a reliable feature-based stereo matching method which determines the more reliable matching pairs earlier and then determines the less reliable ones later using the previously obtained results; and (2) a reliable calibration method for small vertical displacements. The feature point is an edge. The contrast, the position, and the direction of the edge are used to evaluate the matching reliability. Experiments with complicated indoor scenes proved that the proposed method is better than conventional methods.>

Presents an approach to build a world model representation based on planar surfaces from range and video images. The method consists of three sub-processes: (1) to extract the primal surface patches from range and video images; (2) to define the directional uncertainty for the extracted surface patch; and (3) to match the model surfaces of the object expected in the scene with the extracted surface patches using the defined directional uncertainty, model constraints, and a certainty measure of matching, which is defined by directional uncertainty. The preliminary results applied to road scenes are shown.>

An approach is presented for building a 3-D world model from range image sequences derived from road scenes including moving objects. First, a range image is transformed into a height map representing the height information from the assumed ground plane, and then it is segmented into the ground plane and objects on it. In order to capture the resolution and accuracy of the range information and to represent the consistency of the height information between different height maps, the authors define a reliability of the height information for each point on the height map. Using the reliability, the system finds the correspondences of both static and moving objects between different observations, and successively refines the height information and its reliability with newly acquired data, dealing with inconsistent data. The authors show the results obtained using landscape models and a range finder based on the structured light. >

This paper describes an approach to the recognition of stacked objects with planar and curved surfaces. The system works in two phases. In the learning phase, a scene containing a single object is shown one at a time. The range data of a scene are obtained by a range finder. The description of each scene is built in terms of properties of regions and relations between them. This description is stored as an object model. In the recognition phase, an unknown scene is described in the same way as in the learning phase. Then the description is matched to the object models so that the stacked objects are recognized sequentially. Efficient matching is achieved by a combination of data-driven and model-driven search processes. Experimental results for blocks and machine parts are shown.

A structural analysis system for describing natural textures is introduced. The analyzer automatically extracts the texture elements in an input image, measures their properties, classifies them into some distinctive classes (one ``ground'' class and some ``figure'' classes), and computes the distributions of the gray level, the shape, and the placement of the texture elements in each class. These descriptions are used for classification of texture images. An analysis-by-synthesis method for evaluating texture analyzers is also presented. We propose a synthesizer which generates a texture image based on the descriptions. By comparing the reconstructed image with the original one, we can see what information is preserved and what is lost in the descriptions.