When an automatic wheelchair or a self-carrying robot moves along with human agents, prediction for the next possible actions by the participating agents, play an important role in realization of successful cooperation among them. In this paper, we mounted a robot to a wheelchair body so that it provides embodied projective signals to the human agents, indicating the next possible action to be performed by the wheelchair. We have analyzed how human participants, particularly when they are in a multiparty interaction, would respond to such a system in experiments. We designed two settings for the robot's projective behavior. The first design allows the robot to face towards the human agents (Face-to-Face model), and the other allows it to face forward as the human agents do, then turn around to the human agents when it indicates where the wheelchair will move to (Body-Torque model). The analysis examined reactions by the human agents to the wheelchair, his/her accompanying human agent, and others who pass by them in the experiment's setting. The results show that the Body-Torque model seems more effective in enhancing cooperative behavior among the human participants than the Face-to-Face model when they are moving to a forward direction together.

In this paper, we aim to enhance the interaction between the performer and the audience in live idol performances. We propose a system for converting the movements of individual members of an idol group into vibrations and their voices into light on handheld devices for the audience. Specifically, for each performer, the system acquires data on movement and voice magnitudes via an acceleration sensor attached to the right wrist and microphone. The obtained data is then converted into motor vibrations and lights from an LED. The receiving devices for the audience members come in the form of a pen light or doll. A prototype system was made to collect acceleration data and voice magnitude data measurements for our experiments with an idol group in Japan to verify whether the performer's movements and singing voice could be correctly measured during real live performance conditions. We developed a program to present the strength of the movements and singing voice corresponding to one of the members as vibrations and lights based on the information of the recorded data. Then, an experiment was conducted for eight subjects that observed the performance. We found that seven out of eight subjects could identify the idol performer with corresponding vibrations and lighting from the device.

In this work, our goal is to communicate spatial knowledge in Japanese with autonomous robots. We first describe the data collection scheme. We then conduct a study to investigate how Japanese describe spatial relations and what relations they prefer. Based on the observations, we formalize the knowledge by ontologies. With the help of an inference mechanism, our knowledge base is able to store commonsense and discover inexplicit knowledge. We model 16 spatial relations using geometry information. At the language level, we present a natural language interface to execute commands and answer questions. The integrated robotic system unifies visual and spatial information in concert with parsing of semantic interpretation of sentences. Finally, we describe two tasks for validation.

In this paper, we propose a remote monitoring and communication system for elderly care. We aim to detect emergencies concerning elderly people living alone and send alerts to his/her family from a remote location. For this reason, the system has two modules: the monitoring module and the video chat module. When the monitoring module of the system detects any unusual events concerning the elderly, the system sends an alert message to their family's smart phone. Then, their family can monitor the current situation via the video chat module. The elderly and his/her family member can also easily use the video chat function for day-to-day communication without requiring any complex actions for the users. For the video chat display at the elderly side, we use a television set, which all elderly would be expected to have in their rooms. This gives our system a familiar appearance so the elderly can use it without feeling aversion towards the system. The monitoring system for detecting emergencies is a doll-like robot. Besides monitoring, the robot can also initiate simple conversations autonomously in normal situations, which can improve the elderly person's sense of familiarity towards the system.

In this research, we aim to reveal the difference between robot mediated multi-party human interaction and human mediated multi-party human interaction. We do this by examining and comparing the multi-party participants' interactions towards the robot human mediators in regard to the mediator's questions. From this experiment, we found that the participants engaged in interaction among each other more when the questioner was the robot. This paper suggests a concept and a design of robots, which enhances social interaction among humans because of its intervention.

In this article, we discuss our findings from an ethnographic study at an elderly care center where we observed the utilization of two different functions of human gaze to convey service order (i.e., “who is served first and who is served next”). In one case, when an elderly person requested assistance, the gaze of the care worker communicated that he/she would serve that client next in turn. In the other case, the gaze conveyed a request to the service seeker to wait until the care worker finished attending the current client. Each gaze function depended on the care worker's current engagement and other behaviors. We sought to integrate these findings into the development of a robot that might function more effectively in multiple human-robot party settings. We focused on the multiple functions of gaze and bodily actions, implementing those functions into our robot. We conducted three experiments to gauge a combination of gestures and gazes performed by our robot. This article demonstrates that the employment of gaze is an important consideration when developing robots that can interact effectively in multiple human-robot party settings.

The ability to remotely measure heart rate from videos without requiring any special setup is beneficial to many applications. In recent years, a number of papers on heart rate (HR) measurement from videos have been proposed. However, these methods typically require the human subject to be stationary and for the illumination to be controlled. For methods that do take into account motion and illumination changes, strong assumptions are still made about the environment (e.g. background can be used for illumination rectification). In this paper, we propose an HR measurement method that is robust to motion, illumination changes, and does not require use of an environment's background. We present conditions under which cardiac activity extraction from local regions of the face can be treated as a linear Blind Source Separation problem and propose a simple but robust algorithm for selecting good local regions. The independent HR estimates from multiple local regions are then combined in a majority voting scheme that robustly recovers the HR. We validate our algorithm on a large database of challenging videos.

In this paper we describe a prototype remote communication support system to be used as a communication catalyst for dementia care. The system aims to delay the progression of dementia and helps caregivers by facilitating the social interaction of dementia patients. In the system framework, a video telecommunication system was installed in order to enable communication between patients and caregivers. Next, a system control board was designed to maintain robust operation so that users could connect with each other anytime. In the system, we also implemented a reminiscence based therapy application to prevent the progression of dementia. The application can show old photos to dementia patients during a typical video chat. From the caregivers' point of view, our system is also an important form of assistance for repeating announcements and instructions to dementia patients with poor memory. Accordingly, we also made a video replay application which can be set to replay video at a fixed time everyday. Finally, we made aesthetically pleasing robots that serve to make interaction with the system more natural for dementia patients.

We propose a human-robot interaction approach for social robots that attracts and controls the attention of a target person depending on her/his current visual focus of attention. The system detects the person's current task (attention) and estimates the level by using the “task-related contextual cues” and “gaze pattern.” The attention level is used to determine the suitable time to attract the target person's attention toward the robot. The robot detects the interest or willingness of the target person to interact with it. Then, depending on the level of interest of the target person, the robot generates awareness and establishes a communication channel with her/him. To evaluate the performance, we conducted an experiment using our static robot to attract the target human's attention when she/he is involved in four different tasks: reading, writing, browsing, and viewing paintings. The proposed robot determines the level of attention of the current task and considers the situation of the target person. Questionnaire measures confirmed that the proposed robot outperforms a simple attention control robot in attracting participants' attention in an acceptable way. It also causes less disturbance and establishes effective eye contact. We implemented the system into a commercial robotic platform (Robovie-R3) to initiate interaction between visitors and the robot in a museum scenario. The robot determined the visitors' gaze points and established a successful interaction with a success rate of 91.7%.

In this paper, we propose a human-robot interaction system in which the robot detects and classifies the target human's gaze pattern into either spontaneous looking or scene-relevant looking. If the gaze pattern is detected as the spontaneous looking, the robot waits for the target human without disturbing his/her attention. However, if the gaze pattern is detected as the scene-relevant looking, the robot establishes a communication channel with him/her in order to explain about the scene. We have implemented the proposed system into a robot, Robovie-R3 as a museum guide robot and tested the system to confirm its effectiveness.

In order to reduce the burden of caregivers facing an increased demand for care, particularly for the elderly, we developed a system whereby multiple robotic wheelchairs can automatically move alongside a companion. This enables a small number of people to assist a substantially larger number of wheelchair users effectively. This system utilizes an environmental map and an estimation of position to accurately identify the positional relations among the caregiver (or a companion) and each wheelchair. The wheelchairs are consequently able to follow along even if the caregiver cannot be directly recognized. Moreover, the system is able to establish and maintain appropriate positional relations.

In this paper, we propose a robot system able to take visitors on guided tours in a museum. When developing a robot capable of giving a tour of an actual museum, it is necessary to implement a robot system able to measure the location and orientation of the visitors using bearing sensors installed in a specific environment. Furthermore, the robot needs information pertaining to both tour attendees and other visitors in the vicinity, in order to effectively lead a tour for the attendees. Therefore, we propose a new robot system consisting of simple elements such as a laser range finder attached to a pole. By merely placing the sensor poles, we can track the location and orientation of visitors. We employ a group detection method to distinguish tour attendees from other people around the robot. In contrast to previous group detection methods our method can recognize a group of people even when they are not walking. In experiments our system successfully tracked and grouped visitors in the vicinity even when they were just standing.

We are developing a helper robot able to fetch objects requested by users. This robot tries to recognize objects through verbal interaction with the user concerning the objects that it cannot detect autonomously. We have shown that the system can recognize objects based on an ontology for interaction. In this paper, we extend a human description ontology to link a "human description" to "attributes of objects" for our interactive object recognition framework. We develop an interactive object recognition system based on this ontology. Experimental results confirmed that the system could efficiently recognize objects by utilizing this ontology.

We have been conducting research on ways to contribute to reducing the workload of caregivers. We developed a wheelchair capable of following a single accompanying caregiver in a parallel position. However, observations in actual care facilities revealed that people often move as a group: for instance, two caregivers each pushed a wheelchair together. Therefore, we now aim to develop a robotic wheelchair system that allows multiple wheelchairs and accompanying caregivers to coordinate their movement together. However, such a situation necessitates a complex system. Making such a system practical represents an enormous challenge. Therefore, as a first step we decided to break down this complex “multiple-multiple” situation by isolating particular instances with multiple wheelchairs and/or multiple caregivers. In this paper, we focus on the “single-multiple” situation; i.e., a situation where a single wheelchair user is accompanied by multiple accompanies. We propose a robotic wheelchair system that facilitates coordinated movement between the wheelchair and the caregivers. In other words, the wheelchair system will track multiple people in the vicinity using GPGPU, and distinguish the accompanying caregivers from among other passersby based on the trajectory of the individuals' movement.

In this paper, we propose a robot system which can take visitors on guided tours in a museum. When we consider developing a robot capable of giving a tour of an actual museum, we must implement a robot system able to measure the location and orientation of the robot and visitors using bearing sensors installed in a specific environment. Although many previous methods employed markers or tags attached to the robot to obtain positional information through cameras, it is not easy to situate sensors in the environment itself. SLAM is also used to localize the position of the robot. However, its robustness may not be sufficient when the system is deployed in an actual real-life situation since there will be many people moving freely around the robot. On the other hand, the robot needs information pertaining to both tour attendees and other visitors in the vicinity, in order to give a tour for the attendees. Therefore, we propose a new robot system which consists of simple devices such as multiple laser range finders attached to a pole. By just placing the sensor poles, we could track the location and orientation of the robot and visitors at the same time. We then conducted experiments to confirm the effectiveness and accuracy of our system. In addition, we conducted demonstrative experiments that our robot takes three visitors to the guide tour.

We are developing a service robot that provides assisted-care, such as serving tea to the elderly in care facilities. From the sociological analysis of human-human interaction in care facilities, we have found that people often use nonverbal behaviors such as raising their hand to make a request to caregivers. In this paper, we propose a request gesture recognition method for our service robot. In day care facilities, many people are moving around. Hence it is a difficult problem to correctly recognize request gestures among various movements. We have solved this problem by exploiting the sociological interaction analysis result. We have found that a person who would like to make a request to a caregiver waits for the moment when the caregiver looks in the direction towards the person, and then shows a request signal such as raising his/her hand. This means that the robot needs to analyze human motions only when it finds a frontal face in the direction of its face. We have developed a robot system to perform such request recognition and confirmed its effectiveness through experiments.

Recent research in HRI has emphasized the need to design affective interaction systems equipped with social intelligence. A robot's awareness of its social role encompasses the ability to behave in a socially acceptable manner, the ability to communicate appropriately according to the situation, and the ability to detect the feelings of interactive partners, as humans do with one another. In this paper, we propose an intelligent robotic method of attracting a target person's attention in a way congruent to satisfying these social requirements. If the robot needs to initiate communication urgently, such as in the case of reporting an emergency, it does not need to consider the current situation of the person it is addressing. Otherwise, the robot should observe the person to ascertain who or what s/he is looking at (VFOA), and how attentively s/he is doing so (VFOA level). Moreover, the robot must identify an appropriate time at which to attract the target person's attention so as to not interfere with his/her work. We have realized just such a robotic system by developing computer vision methods to detect a target person's VFOA and its level, and testing the system's effectiveness in a series of experiments.

Although it is desirable for wheelchair users to move alone by operating wheelchairs themselves, they are often pushed or accompanied by caregivers, especially in elderly care facilities. In designing robotic wheelchairs, therefore, it is important to consider how to reduce the caregiver's load and support their activities. We particularly focus on communication among wheelchair users and caregivers because face-to-face communication is known to be effective in ameliorating elderly mental health issues. Hence, we propose a maneuverable robotic wheelchair that can move autonomously towards a goal while considering the situation of the accompanying caregiver. Our robotic wheelchair starts to move autonomously towards the goal as soon as the caregiver sets the goal position, such as a bathroom, by utilizing its touch panel interface. While moving towards the goal, the robotic wheelchair autonomously controls its speed and direction of movement to facilitate natural communication between the wheelchair user and the caregiver, by observing the situation of the accompanying caregiver.

Museum guide robots should observe visitors in order to identify those who may desire a guide, and then initiate conversation with them. This paper presents a model for such robot behavior. Initiation of conversation is an important concern for social service robots such as museum guide robots. When people enter into a social interaction, they tend to situate themselves in a spatial-orientational arrangement such that each is facing inward around a space to which each has immediate access. When this kind of particular spatial formation occurs, they can feel that they are participating in the conversation; once they perceive their participation, they will subsequently try to maintain this spatial formation. We developed a model that describes the constraints and expected behaviors in initiation of conversation. Experimental results demonstrate that our model significantly improves a robot's performance in initiating conversation.

A major challenge is to design a robot that can attract and control human attention in various social situations. If a robot would like to communicate a person, it may turn its gaze to him/her for eye contact. However, it is not an easy task for the robot to make eye contact because such a turning action alone may not be enough in all situations, especially when the robot and the human are not facing each other. In this paper, we present an attention control approach through robot's behaviors that can attract a person's attention by three actions: head turning, head shaking, and uttering reference terms corresponding to three viewing situations in which the human vision senses the robot (near peripheral field of view, far peripheral field of view, and out of field of view). After gaining attention, the robot makes eye contact through showing gaze awareness by blinking its eyes, and directs the human attention by eye and head turning behaviors to share an object.

From our ethnographic studies on various kinds of museums, we discovered that guides routinely propose questions to visitors in order to draw their attention towards both his/her explanation and the exhibit. The guides' question sequences tend to begin with a pre-question which serves to not only monitor visitors' behavior and responses, but to also alert visitors that a primary question would follow. We implemented this questioning-strategy with our robot system and investigated whether this strategy would also work in human-robot interaction. We developed a vision system that enables the robot to choose an appropriate visitor by monitoring a visitor's response from the initiation of a pre-question to the following pause. Results indicate that this questioning-strategy works effectively in human-robot interaction. In this experiment, the robot asked visitors about a photograph. At the pre-question, the robot delivered a rather easy question followed by a more challenging question (Figure 1). More participants turned their head away from the exhibition when they were not sure about their answer to the question. They either faced away from the robot, or smiled wryly at the robot or at each other. These types of behaviors index participants' states of knowledge, which we could utilize to develop a system by which the robot could choose an appropriate candidate by computational recognition.

In this paper, we present our work designing a robot that explains an exhibit to multiple visitors in a museum setting, based on ethnographic analysis of interactions between expert human guides and visitors. During the ethnographic analysis, we discovered that expert human guides employ some identical strategies and practices in their explanations. In particular, one of these is to involve all visitors by posing a question to an appropriate visitor among them, which we call the "creating a puzzle" sequence. This is done in order to draw visitors' attention towards not only the exhibit and but also the guide's explanation. While creating a puzzle, the human guide can monitor visitors' responses and choose an "appropriate" visitor (i.e. one who is likely to provide an answer). Based on these findings, sociologists and engineers together developed a guide robot that coordinates verbal and non-verbal actions in posing a question or "a puzzle" that will draw visitors' attention, and then explain the exhibit for multiple visitors. During the explanation, the robot chooses an "appropriate" visitor. We tested the robot at an actual museum. The results show that our robot increases visitors' engagement and interaction with the guide, as well as interaction and engagement among visitors.

A mobile museum guide robot is expected to establish a proper spatial formation with the visitors. After observing the videotaped scenes of human guide-visitors interaction at actual museum galleries, we have developed a mobile robot that can guide multiple visitors inside the gallery from one exhibit to another. The mobile guide robot is capable of establishing spatial formation known as "F-formation" at the beginning of explanation. It can also use a systematic procedure known as "pause and restart" depending on the situation through which a framework of mutual orientation between the speaker (robot) and visitors is achieved. The effectiveness of our method has been confirmed through experiments.

Recently, several robotic/intelligent wheelchairs equipped with user-friendly interfaces and/or autonomous functions to fulfill set goals have been proposed. Although wheelchair users may wish to go out alone, caregivers often accompany them. Therefore, it is important to consider how to reduce the caregivers' load while supporting their activities and facilitating communication between the caregiver and wheelchair user. Hence, we have proposed a robotic wheelchair that can move alongside a caregiver by observing his/her behavior, such as body position and orientation. This form of motion enables easy communication between the wheelchair user and the caregiver. However, in our previous system, the caregiver had to set their own initial position by tapping the touchscreen displaying the distance data image captured by a laser range sensor. In this paper, therefore, we introduce a new interface we designed by incorporating an omni-directional camera so that the accompanying caregiver can easily set the initial position by tapping his/her facial image instead. Moreover, when the system loses the accompanying caregiver due to them making an unexpected motion, it promptly re-identifies the caregiver by using color appearance cues. These functions improve the usefulness of our robotic wheelchair and heighten the possibility of our wheelchair being utilized in our daily life.

A major challenge in HRI is to design a social robot that can attract a target human's attention to control his/her attention toward a particular direction in various social situations. If a robot would like to initiate an interaction with a person, it may turn its gaze to him/her for eye contact. However, it is not an easy task for the robot to make eye contact because such a turning action alone may not be enough to initiate an interaction in all situations, especially when the robot and the human are not facing each other or the human intensely attends to his/her task. In this paper, we propose a conceptual model of attention control with four phases: attention attraction, eye contact, attention avoidance, and attention shift. In order to initiate an attention control process, the robot first tries to gain the target participant's attention toward it through head turning, or head shaking action depending on the three viewing situations where the robot is captured in his/her field of view (central field of view, near peripheral field of view, and far peripheral field of view). After gaining her/his attention, the robot makes eye contact only with the target person through showing gaze awareness by blinking its eyes, and directs her/his attention toward an object by turning its eyes and head cues. Moreover, the robot can show attention to aversion behaviors if non-target persons look at it. We design a robot based on the proposed approach, and it is confirmed as effective to control the target participant's attention in experimental evaluation.

A social robot needs to attract the attention of a target human and shift it from his/her current focus to what is sought by the robot. The robot should recognize the current target's attention level to smoothly perform this attention control. In this paper, we propose a vision-based system to detect the level of attention or willingness of the target person towards the robot and to control his/her attention. The system estimates the attention level from rich visual cues of human's face and head. Then, by timing target's attention to determine the appropriate attention level, it generates aware signals and makes eye contact with the target. Finally, the robot shifts the target's attention to an intended direction. The experimental results reveal that the proposed system is effective in controlling the target's attention.

It is a major challenge in HRI to design a social robot that is able to direct a target human's attention towards an intended direction. For this purpose, the robot may first turn its gaze to him/her in order to establish eye contact. However, such a turning action of the robot may not in itself be sufficient to make eye contact with the target person in all situations, especially when the robot and the person are not facing each other or the human is intensely engaged in a task. In this paper, we propose a conceptual model of attention control with five phases: attention attraction, eye contact, attention avoidance, gaze back, and attention shift. We conducted two experiments to validate our model in human-robot interaction scenarios.

This paper presents our ongoing work developing service robots that provide assisted-care, such as serving tea to the elderly in care facilities. In multi-party settings, a robot is required to be able to deal with requests from multiple individuals simultaneously. In particular, when the service robot is concentrating on taking care of a specific person, other people who want to initiate interaction may feel frustrated with the robot. To a considerable extent this may be caused by the robot's behavior, which does not indicate any response to subsequent requests while preoccupied with the first. Therefore, we developed a robot that can display acknowledgement, in a socially acceptable manner, to each person who wants to initiate interaction. In this paper we focus on the task of tea-serving, and introduce a robot able to bring tea to multiple users while accepting multiple requests. The robot can detect a person's request (raising their hand) and move around people using its localization system. When the robot detects a person's request while serving tea to another person, it displays its acknowledgement by indicating "Please wait" through a nonverbal action. Because it can indicate its acknowledgement of their requests socially, people will likely feel more satisfied with the robot even when it cannot immediately address their needs.

This video presents our ongoing work developing a robotic wheelchair that can move automatically alongside a caregiver. Recently, several robotic/intelligent wheelchairs possessing autonomous functions for reaching a goal and/or user-friendly interfaces have been proposed. Although ideally wheelchair users may wish to go out alone, they are often accompanied by caregivers. Therefore, it is important to consider how to reduce the caregivers' load and support their activities and facilitate communication between the wheelchair user and caregiver. Moreover, a sociologist pointed out that when a wheelchair user is accompanied by a companion, the latter is inevitably seen as a caregiver [1]. In other words, the equality of the relationship is publicly undermined when the wheelchair is pushed by a companion. Hence, we propose a robotic wheelchair which can move alongside a caregiver or companion, and facilitate easy communication between them and the wheelchair user. However, it is not always desirable for a caregiver to be alongside a wheelchair. For instance, a caregiver may step in front of the wheelchair to open a door, and pedestrians may be encumbered by the wheelchair and companion if they move along side-by-side in a narrow corridor. To cope with these problems, our robotic wheelchair can move alongside a caregiver collaboratively depending on the circumstances. A laser range sensor is employed to track the caregiver and observe the environment around the wheelchair [2]. When obstacles are detected in the wheelchair's path of motion, it adjusts its position accordingly. In the video we demonstrate these functions of our robotic wheelchair. We are now conducting experiments to confirm the effectiveness of our wheelchair at an elderly care center in Japan.

This paper introduces a service robot that provides assisted-care, such as serving tea to the elderly in care facilities. In multi-party settings, a robot is required to be able to deal with requests from multiple individuals simultaneously. In particular, when the service robot is concentrating on taking care of a specific person, other people who want to initiate interaction may feel frustrated with the robot. To a considerable extent this may be caused by the robot's behavior, which does not indicate any response to subsequent requests while preoccupied with the first. Therefore, we developed a robot that can project the order of service in a socially acceptable manner to each person who wishes to initiate interaction. In this paper we focus on the task of tea-serving, and introduce a robot able to bring tea to multiple users while accepting multiple requests. The robot can detect a person raising their hand to make a request, and move around people using its mobile functions while avoiding obstacles. When the robot detects a person's request while already serving tea to another person, it projects that it has received the order by indicating “you are next” through a nonverbal action, such as turning its gaze to the person. Because it can project the order of service and indicate its acknowledgement of their requests socially, people will likely feel more satisfied with the robot even when it cannot immediately address their needs. We confirmed the effectiveness of this capability through an experiment in which the robot distributed snacks to participants.

This paper presents a method of selecting the answerer from audiences for a museum guide robot. We performed the preliminary experiments that a robot distributed its gaze towards visitors to select an answerer and analyzed visitors' responses. From these experiments, we have found that the visitors who are asked questions by the robot feel embarrassed when they have no prior knowledge about the question and the visitor's gaze during the question plays an important role to avoid being asked question. Based on these findings we developed functions for a guide robot to select the answerer by observing behaviors of multiple visitors. Multiple visitors' head motions are tracked and recognized by using an omni-directional camera and a laser range sensor. The robot detects the visitors' positive and negative responses by observing their head motions while asking questions. We confirmed the effectiveness of our method by experiments.

Object category detection with large appearance variation is a fundamental problem in computer vision. The appearance of object categories can change due to intra-class variability, viewpoint, and illumination. For object categories with large appearance change a sub-categorization based approach is necessary. This paper proposes a sub-category optimization approach that automatically divides an object category into an appropriate number of sub-categories based on appearance variation. Instead of using a predefined intra-category sub-categorization based on domain knowledge or validation datasets, we divide the sample space by unsupervised clustering based on discriminative image features. Then the clustering performance is verified using a sub-category discriminant analysis. Based on the clustering performance of the unsupervised approach and sub-category discriminant analysis results we determine an optimal number of sub-categories per object category. Extensive experimental results are shown using two standard and the authors' own databases. The comparison results show that our approach outperforms the state-of-the-art methods.

In this paper, we report on our development of a robotic system that assists people in accomplishing simple tasks in daily life (e.g., retrieving objects for handicapped and elderly people). These tasks, inevitably involve detecting various kinds of objects. In particular, here, we present an interactive method to detect objects using spatial information. Our experimental results confirm the usefulness and efficiency of our system. We also show how the approach can be improved and highlight necessary directions for future research.

This paper presents a method in a museum guide robot to choose an appropriate answerer among multiple visitors. First, we observed and videotaped scenes of gallery talk when human guides ask visitors questions. Based on an analysis of this video, we have found that the guides selects an answerer by distributing his or her gaze towards multiple visitors and observing the visitors' gaze responses during the question. Then, we performed experiments on a robot that distributes its gaze towards multiple visitors, and analyzed visitors' responses. From these experiments, we found that visitors who are asked questions by the robot felt embarrassed when he or she had no prior knowledge about the questions, and that visitor gaze during the questions plays an important role in avoiding being asked questions. Based on these findings, we have developed a function in a guide robot that observes visitor gaze response and selects an appropriate answerer based on these responses. Gaze responses are tracked and recognized using an omnidirectional camera and a laser range sensor. The effectiveness of our method was confirmed through experiments.

Designing technologies that support the explanation of museum exhibits is a challenging domain. In this paper we develop an innovative approach - providing a robot guide with resources to engage visitors in an interaction about an art exhibit. We draw upon ethnographical fieldwork in an art museum, focusing on how tour guides interrelate talk and visual conduct, specifically how they ask questions of different kinds to engage and involve visitors in lengthy explanations of an exhibit. From this analysis we have developed a robot guide that can coordinate its utterances and body movement to monitor the responses of visitors to these. Detailed analysis of the interaction between the robot and visitors in an art museum suggests that such simple devices derived from the study of human interaction might be useful in engaging visitors in explanations of complex artifacts.

Recently, several robotic/intelligent wheelchairs have been proposed that employ user-friendly interfaces or autonomous functions. Although it is often desirable for user to operate wheelchairs on their own, they are often accompanied by a caregiver or companion. In designing wheelchairs, it is important to reduce the caregiver load. In this paper we propose a robotic wheelchair that can move with a caregiver side by side. In contrast to a front-behind position, in a side-by-side position it is more difficult for wheelchairs to adjust when the caregiver makes a turn. To cope with this problem we present a visual-laser tracking technique. In this technique, a laser range sensor and an omni-directional camera are integrated to observe the caregiver. A Rao-Blackwellized particle filter framework is employed to track the caregiver's position and orientation of both body and head based on the distance data and panorama images captured from the laser range sensor and the omni-directional camera. After presenting this technique, we introduce an application of the wheelchair for museum visit use.

The main focus of this special issue is to bring together solutions from research and engineering for the automatic understanding of a complex scene, via a multimodal array of sensors with automation of adaptation as the theme connecting various research areas. Four papers comprise this special issue.

As research over the last several decades has shown that non-verbal actions such as face and head movement play a crucial role in human interaction, such resources are also likely to play an important role in human-robot interaction. In developing a robotic system that employs embodied resources such as face and head movement, we cannot simply program the robot to move at random but rather we need to consider the ways these actions may be timed to specific points in the talk. This paper discusses our work in developing a museum guide robot that moves its head at interactionally significant points during its explanation of an exhibit. In order to proceed, we first examined the coordination of verbal and non-verbal actions in human guide-visitor interaction. Based on this analysis, we developed a robot that moves its head at interactionally significant points in its talk. We then conducted several experiments to examine human participant non-verbal responses to the robot's head and gaze turns. Our results show that participants are likely to display non-verbal actions, and do so with precision timing, when the robot turns its head and gaze at interactionally significant points than when the robot turns its head at not interactionally significant points. Based on these findings, we propose several suggestions for the design of a guide robot.

In this paper we consider the development of a museum guide robot that has both autonomous and remotely controlled features. We focus on the capabilities such a robot could have to help focus the attention of a visitor on an object or artefact. Inspired by studies of social interaction, which investigate whether the robot could deploy "restarts" and "pauses" at certain moments in its talk to first elicit the visitor's attention/gaze towards the robot. We report an experiment where we deployed such a robot to interact with real visitors to a science museum. These experiments show that such a strategy does seem to have a significant impact on obtaining the visitor's gaze.

Nonverbal behavior plays an important role in human communication. In this paper, we propose a novel museum guide robot that has three different types of communication modes (autonomous, remote-control, and remote-interactive), which are integrated to interact with visitors through nonverbal behavior. First, the autonomous mode is an autonomously controlled mode in which the robot can directly interact with visitors through nonverbal behaviors such as head movements or arm gestures. Second, when a visitor begins to ask the robot questions, the communication is changed into a remote-control mode. In this mode, a remote operator controls the robot with gestures to interact with visitors. Third, in the remote-interactive mode, a remote operator can also interact with the robot with voice and gestures through a network. In particular, the remote operator can ask the robot to do something with nonverbal behaviors such as pointing while saying, “Bring this brochure to him.” These three communication modes are switched seamlessly depending on the situation.

Service robots need to be able to recognize and identify objects located within complex backgrounds. Since no single method may work in every situation, several methods need to be combined. However, there are several cases when autonomous recognition methods fail. We propose an interactive recognition method in these cases. To develop a natural Human Robot Interaction (HRI), it is necessary that the robot should unambiguously perceive the description of an object given by human. This paper reports on our experiment in which we examined the expressions humans use in describing ordinary objects. The results show that humans typically describe objects using one of multiple colors. The color is usually either that of the object background or that of the largest object portion. Based on these results, we describe our development of a robot vision system that can recognize objects when a user adopts simple expressions to describe the objects. This research suggests the importance of connecting ‘symbolic expressions’ with the ‘real world’ in human-robot interaction.

We are currently working on a museum guide robot with an emphasis on "friendly" human-robot interaction displayed through nonverbal behaviors. In this paper, we focus on head gestures during explanations of exhibits. The outline of our research is as follows. We first examined human head gestures through an experimental, sociological approach. From this research, we have discovered how human guides coordinate their head movement along with their talk when explaining exhibits. Second, we developed a robot system based on these findings. Third, we evaluated human-robot interaction, again using an experimental, sociological approach, and then modified the robot based on the results. Our experimental results suggest that robot head turning may lead to heightened engagement of museum visitors with the robot. Based on our preliminary findings, we will describe a museum guide robot that first works autonomously and, if necessary, can turn into remote-control mode operated by a human to engage in more complex interaction with visitors.

We are currently working on a museum guide robot with an emphasis on "friendly" human-robot interaction as displayed through nonverbal behaviors. In this paper, we focus on head gestures during explanations of exhibits. The outline of our research is as follows. We first examined human head gestures through an experimental, sociological approach, and have discovered the ways human guides coordinate their head movement along with their talk when explaining exhibits. Second, we developed a robot system based on these findings. Third, we evaluated human-robot interaction, again using an experimental, sociological approach, and then modified the robot based on the results. Our experimental results suggest that robot head turning may lead to heightened engagement of museum visitors.

Face or head movement plays an important role in human communication. This paper presents a museum guide robot that moves its head to communicate smoothly with humans. We have analyzed the behavior of human guides when they explain exhibits to visitors. Then, we have developed a robot system that can recognize the human's face movement using vision. The robot turns its head depending on the human's face direction and the contents of utterances. We use the analysis results of human behavior to control the head movements. Experimental results show that it is effective for the guide robot to turn its head while explaining exhibits

An effective human-robot interaction is essential for wide penetration of service robots into the market. Such robots need vision systems to recognize objects. It is, however, difficult to realize vision systems that can work in various conditions. More robust techniques of object recognition and image segmentation are essential. Thus, we have proposed to use the human user's assistance for object recognition through speech. The robot asks a question to which the user can easily answer and whose answer can efficiently reduce the number of candidate objects even if there are occluded objects and/or objects composed of multicolor parts in the scene. It considers the characteristics of features used for object recognition such as the easiness for humans to specify them by word, thus generating a user-friendly and efficient sequence of questions. Experimental results show that the robot can detect target objects by asking the questions generated by the method.

This paper deals with a basic dance training system with multimodal information presentation. The system targets on beginners and enables them to learn basics of dances easily. Watching a video showing the performance of dance masters is one of the most effective ways of learning dances. However, it is difficult for dance beginners to obtain useful information well only by watching video. They can hardly know timing information and the amount of translation. Although we can tell how to move our arms or legs from video, it is difficult to know when to start moving them. In addition, they cannot obtain translational motion, especially that in the depth direction (how far the dancers move forward or backward). We solve the first issue by developing active devices. The active devices consist of some vibro-motors and are designed to direct action-starting cues with vibration. Concerning the second issue, the image display on a mobile robot is introduced into the system. We can learn the amount of translation just by following the robot. Experimental results show the effectiveness of our proposed method.

Eye contact is an effective means of controlling human communication, such as in starting communication. It seems that we can make eye contact if we simply look at each other. However, this alone does not establish eye contact. Both parties also need to be aware of being watched by the other. We propose a method of two-way eye contact for human-robot communication. When a human wants to start communication with a robot, he/she watches the robot. If it finds a human looking at it, the robot turns to him/her, changing its facial expressions to let him/her know its awareness of his/her gaze. When the robot wants to initiate communication with a particular person, it moves its body and face toward him/her and changes its facial expressions to make the person notice its gaze. We show several experimental results to prove the effectiveness of this method. Moreover, we present a robot that can recognize hand gestures after making eye contact with the human to show the usefulness of eye contact as a means of controlling communication.

This paper presents a basic dance training system with a mobile robot. The system targets on beginners and enables them to learn basics of traditional dances easily. Learners can observe not only a fixed screen in front of them but also the display mounted on a mobile robot that moves according to the dance steps. These displays show actions of a dance master and real-time captured movement of the learners. The robot always takes its appropriate position so that the learners can easily know bad points of their actions. In other words, the robot plays a role as a "mobile mirror." We verified the effectiveness of the proposed method and system through experiments.

This paper presents a vision-speech system for service robots that can learn the user's customs and objects fixed in the environment while helping the user, and can perform their tasks more efficiently with less user's burden. We are working on a service robot that fetches the objects ordered by the user through speech. The robots need vision to recognize the objects. It asks the user for help by speech if its vision fails. In early stages, it asks the user for help many times and requires time to detect the objects by vision, because it does not know much about the environment. We assume that humans usually put a particular object on a small number of places. The robot obtains such information and the environmental knowledge while accomplishing the user's demands. In later stages, the user does not need to say details because the robot knows where the objects usually are. Moreover, the vision processing time is greatly reduced because it knows what operations can work in such cases. Experiments using a robot system show the usefulness of the proposed system.

This paper presents a display system that enables us to find necessary information quickly. Multiple windows containing information appear from the bottom of the display, moving upward swiftly. We move our eyes to obtain the information in the windows. The speed of the eye movements controls the upward moving speed of the windows so that we can obtain information from the system as fast as possible. If we find anything interesting in a window and stop the eye movements to look at it more carefully, the system stops the window motion, enlarging the window to show details.

This paper presents a user interface for a service robot that can bring the objects asked by the user. Speech-based interface is appropriate for this application, but it alone is not sufficient. The system needs a vision-based interface to recognize gestures as well. Moreover, it needs vision capabilities to obtain the real world information about the objects mentioned in the user's speech. For example, the robot needs to find the target object ordered by speech to carry out the task. This can be considered as a vision assisted speech. However, vision sometimes fails to detect the objects. Moreover, there are objects for which vision cannot be expected to work well. In these cases, the robot tells the current status to the user so that he/she can give advice by speech to the robot. This can be considered as a speech assisted vision through the user. This paper presents how the mutual assistance between speech and vision works and demonstrates promising results through experiments.

With the increase in the number of senior citizens, there is a growing demand for human-friendly wheelchairs as mobility aids. One of the main issues in the robotic wheelchair research is autonomous obstacle avoidance for safety. However, this is difficult because most moving obstacles in the real world are human beings. They sometimes change their motion abruptly. The paper presents an intelligent wheelchair that can avoid collisions with such human pedestrians safely and comfortably for each other. We assume that the information whether or not a pedestrian has noticed the wheelchair and which direction he/she wants to go can appear in the face direction. Thus our intelligent wheelchair is continuously observing the pedestrian's face in its front area, realizing smooth passing by changing its collision avoidance strategy based on the face information and the range data from the ultrasonic sensors. Experimental results show the effectiveness and comfortableness of the proposed method.

We presented an intelligent wheelchair whose motion can be controlled by the user's face direction. We propose to add intelligence to our wheelchair when the user is not riding. It can recognize the user's face and can move according to the gestures made by the user. Gesture is a good means to give commands because it can be used in noisy conditions and when we would not like to speak loudly in public places. However, environments where wheelchairs are used cannot be controlled. This makes gestures recognition difficult. We propose an interactive way to solve this problem. When the wheelchair is not certain about the meaning of the user's gesture, it guesses the meaning and moves a little accordingly to show its guess to the user. Then it observes the user's response, judging whether its guess is correct or not. This guess-action-observation cycle is repeated until the wheelchair can understand the user's intention.

This paper proposes a method for the rapid and precise estimation of human hand postures by combining 2D appearance matching and 3D model-based fitting. First a rough posture estimate is obtained by image indexing. Each possible hand appearance generated from a given 3D shape model is labeled by an index obtained by PCA compression and registered with its 3D model parameters in advance. By retrieving the index of the input image, the method can obtain the matched appearance image and its 3D parameters rapidly. Then, starting from the obtained rough estimate, it estimates the posture and moreover refines the given initial 3D model by model-fitting. The sequential activation of the two processes in every frame gives the precise posture estimate rapidly. The effectiveness of the method is shown by experimental results.

Humans can communicate with each other by gestures. This paper presents a robot system with such capability. The robot detects its user by recognizing his/her face. It can accept his/her commands given by gestures. The user may use unknown gestures to the robot. If the robot does not respond to his/her gesture, the user usually iterates the same gesture. The robot detects this repetitive pattern as an intentional gesture by which the user wants to give it some order. Then it shows a little action according to the meaning of the gesture that the robot guesses. It observes the user's reaction to its action. If he/she continues the same pattern gesture, the robot considers that its understanding is right, completing the action. It also registers the pattern as a gesture of the guessed meaning. Otherwise, it iterates the same procedure by taking another action as a candidate of the meaning. We have implemented such interactive capability on our intelligent wheelchair.

With the increase in the number of senior citizens, there is a growing demand for human-friendly wheelchairs as mobility aids. One of the main issues in robotic wheelchair research is autonomous obstacle avoidance for safety. However, this is difficult because most of the moving obstacles in the real world are human beings, and they sometimes change their motion abruptly. This paper presents an intelligent wheelchair that can avoid collisions with such pedestrians (and each other) safely and comfortably. We assume that the information as to whether or not a pedestrian has noticed the wheelchair and which direction he/she wants to go can appear in the direction of the pedestrian's face. Thus, our intelligent wheelchair continuously observes the pedestrian's face in its frontal area, realizing smooth passing by changing its collision-avoidance strategy based on the face information and the range data from ultrasonic sensors. The effectiveness of the proposed method is shown by a comparative experiment with a method using only ultrasonic sensors.

Human interfaces are usually designed to respond only to intentional human behaviors. However, humans show unintentional behaviors as well. They can convey useful information to realize user-friendly human interfaces. This paper presents how to combine observations of both types of behaviors by taking two human- machine systems: a gesture-based interface and an intelligent wheelchair. In the first system, intentional hand gestures are chosen using unintentional behaviors. In the second system, near unintentional behaviors following intentional behaviors can be used to control the wheelchair motion. Experimental systems working in real time have been developed. Operational experiments prove our approach promising.

With the increase of senior citizens, there is a growing demand for human-friendly wheelchairs as mobility aids. The paper proposes the concept of a robotic wheelchair to meet this need. It has three motion control modes. One is an autonomous control mode. The wheelchair has autonomous navigation capabilities, such as avoiding obstacles and moving straight toward a target object or along a corridor using ultrasonic and vision sensors. Another is a manual control mode with minimum intentional operations. The wheelchair observes the user's face by vision, moving in the direction indicated by the use's face movements. The third is a remote control mode. When the user is not riding, it can find him/her using face recognition. The user can make it come or go by hand gestures. We have developed an experimental robotic wheelchair based on the above concept. Experimental results prove our approach promising.

With the increase in the number of senior citizens, there is a growing demand for human-friendly wheelchairs as mobility aids. To meet this need, we proposed a robotic wheelchair which can be controlled by turning our face in the direction where we would like to go. Although it can be used easily, there is a problem that unintentional movements of our face may interfere with the wheelchair motion. The paper presents our new version of the wheelchair improved by observing both user and environment. It effectively integrates autonomous capabilities and the interface by face direction. It uses the sensor information obtained for autonomous navigation to solve the problem with the control by face direction. Also, if it can understand the user's intentions from observing the face, it chooses an appropriate autonomous navigation function to reduce the user's burden of operation.

We need a reliable visual surveillance system that can work under various disturbances, such as changes of lighting and movements of background objects. We propose a method exploiting information over space and time. First, optical flow is computed, and then regions with uniform flow are extracted. The regions are hypothesized as parts of detection targets. Assuming that the targets move in constant directions at constant speeds, we predict a path of each region and vote it in the space-time cube. If enough number of votes support a particular path, the method reports the detection of the intruder. We have implemented this method using multiple DSP boards and constructed a system that works in real time. Experimental results show that the system can detect intruders walking with many objects moving in their backgrounds.

Proposes a method of removing ambiguities in robot tasks by a multimodal human-robot interface consisting of verbal and nonverbal communication. Such ambiguities often arise from failures of the robot vision system. However, it is not easy to solve this problem only by improving computer vision techniques. Thus, our robot asks a human such a question that a natural reply to it will contain helpful information to adapt the vision system for the current situation. We present a robot system that can bring the object ordered by a human by verbal and nonverbal behaviors.

With the increase of senior citizens, there is a growing demand for human-friendly wheelchairs as mobility aids. The paper proposes a concept of an intelligent wheelchair to meet this need. It can understand human intentions by observing the user's nonverbal behaviors and can move in accordance with the user's wish with minimum human operations. The paper also describes our experimental robotic wheelchair system. Human intentions appear most on the face. Thus, the experimental system observes the human face, computing its direction. As the first step toward the intelligent wheelchair, we have made experiments on controlling the system's motion by the face direction. Experimental results prove our approach promising.

This paper proposes a method of removing ambiguities in robot tasks by a multimodal human-robot interface consisting of verbal and nonverbal communication. Such ambiguities often arise from failures of robot vision systems. However, it is not easy to solve this problem only by improving computer vision techniques. Thus, our robot asks a human such a question that a natural reply to it will contain helpful information to adapt the vision system to the current situation. We present a robot system that can bring objects ordered by a human through verbal and nonverbal behaviors.

Describes a method of planning of observation and motion for a mobile robot to interpret a road intersection scene and to reach the intersection as soon as possible considering the uncertainty of interpretation. From a monocular color image, candidate regions are extracted for objects which are related to the intersection type. The probabilities of the regions coming from the objects are calculated from probabilistic models of the objects. From the probabilities of the regions and the relation between the intersection type and the objects, the current probability of intersection types is calculated. If the intersection type is ambiguous, the robot plans the observation and motion which minimize the expectation of the cost (time) to complete the task. The robot takes these actions and tries to determine the intersection type again. This process is iterated until the task is completed. The experimental result is shown for an actual intersection scene.

Presently, mobile robots are navigated by means of a number of methods, using navigating systems such as the sonar-sensing system or the visual-sensing system. These systems each have their strengths and weaknesses. To fully utilise the strengths of both the sonar and visual sensing systems, this paper proposes a fusion of navigating methods involving both the sonar and visual systems as primary sources to produce a fast, efficient and reliable obstacle-avoiding and navigating system. Furthermore, to further enhance a better perception of the surroundings and to improve the navigation capabilities of the mobile robot, active sensing modules are also included. The result is an active sensor fusion system for the collision avoiding behaviour of mobile robots, This behaviour can then be incorporated into other purposive behaviours (e.g. object seeking, path finding etc.). The validity of this system is also shown in real robot experiments.

The parametric eigenspace method was proposed by Murase-Nayar (1995) to recognize objects and their poses. It could be applied to robot navigation to locate the robot position. However, since similar images may often be taken at multiple locations in real scenes, it cannot always give the robot position reliably with a single image input. This problem can be solved using active vision, that is, combining localization results for images taken at multiple camera positions. Since similar images are projected to points close to one another in the eigenspace, we can tell before actual navigation when we cannot expect reliable localization results with a single image by examining the eigenspace. Moreover, further analysis of the eigenspace can give the best action sequences of camera motion to efficiently localize the robot position. This paper presents such an eigenspace analysis method. Experimental results show the effectiveness of the method.

We need a reliable moving object detection method that can work under various disturbances, such as changes of lighting and movements of background objects, to realize practical automatic surveillance systems. Using local information alone could not achieve this goal. Hence, we propose a method exploiting information over space and time. Optical flow is first computed, and then regions with uniform flow are extracted. The regions are hypothesized as parts of detection targets. Assuming that the targets pass in the scene at constant speeds, we predict the path of each region, voting it in the space-time cube as in the Hough transform. If enough number of votes support a particular path, the method reports the detection of a moving object. Experimental results show that it can detect walking people in complex scenes with moving background objects, such as trees shaking in the wind.

This paper describes a method of determining contours of moving objects in a cluttered scene by integrating optical flow and edges. If the motion of an object is similar to that of the background, the contour is not determined only by optical flow. If the background of a scene is cluttered, the contour is not determined only from edges because many edges may be extracted in the background and no edges may be extracted on some parts of the contour. In the proposed method, the contour is determined by using optical flow and edges in a long sequence. The whole contour of a moving object is eventually obtained by accumulating edges near motion boundaries over an image sequence. The method can also determine the occlusion relation of two overlapping objects by checking if edges exist on the predicted contours of objects. Experimental results for synthetic and real images show the usefulness of the method.

This paper describes a method of recognition of intersection scenes by attentive observation considering the uncertainty of recognition, for mobile robots on roads. From a monocular color image, homogeneous color regions are extracted. Probabilities of the regions coming from specific objects are calculated. From these probabilities and the relationship between these objects and intersection types, the current probability distribution of intersection types is calculated. If the entropy of the probability distribution is lower than a certain threshold, the robot adopts the best hypothesis. Otherwise, the robot selects and observes the part which can minimize the expectation of the entropy attentively. These actions are iterated until the entropy becomes lower than the threshold. The experimental results are shown for actual intersection scenes including a white mark, a curve mirror and so on.

Geometric invariants from multiple views provide useful information for 3D object recognition. However, conventional object recognition methods using invariants based on point features cannot achieve efficient recognition because of large amount of combinations of point features in invariant calculation. To avoid this problem, the authors propose to use more complex features. They adopt arrow junctions and conics as complex features because man-made objects have often trihedral polyhedra (eg. parallelepiped) and circles and they make arrow junctions and conics in images, respectively. The multiple view affine invariance theory can be directly used for arrow junctions. For conics, they propose two types of invariants. They have developed an object recognition method exploiting these invariants. In addition to the recognition method with two input images, they propose a recognition method that needs only a single input image by substituting an image of a target object stored in the model library. Experimental results using 240 pair of images for 24 objects confirm the usefulness of the methods.

This paper presents a human-robot interface system that enables a user to move a robot by moving his hand. The system adopts uncalibrated stereo vision based on the affine invariants from multiple views. The system can interpret hand gestures both in the user-centered frame and in the world-fixed frame. Suppose the user wants to indicate the direction of a mobile robot motion by the direction of his hand. If he moves his hand forward, the robot moves straight ahead regardless of his body position, that is, regardless of the hand direction in the world by the former interpretation. By the latter interpretation, however, the robot turns towards the direction indicated by the hand. The former is useful when we operate a teleoperation robot while watching the images sent from the robot. Operation experiments show the usefulness of the system.

This paper presents a vision-based human-computer interface system that enables the user to move a target object in 3D CG world by moving his/her hand. Unlike conventional systems, the system does not need any camera calibration. In addition, it uses a user-centered frame so that the user can control the object by hand motions matched with human intuition. For example, the user can move the object forward by moving his/her hand forward even if he/she has changed his/her body position. The system is based on the multiple view affine invariance theory. It calculates hand positions as invariant coordinates in the basis derived from four points on the user's body. Actual system implementation and experimental results show the usefulness of the system.>

An efficient and geometrically intuitive algorithm for reliably interpreting the image velocities of moving objects in 3-D is presented. It is well known that under a weak perspective the image motion of points on a plane can be characterized by an affine transformation. It is shown that the relative image motion of a nearby non-coplanar point and its projection on the plane is equivalent to motion parallax, and because it is independent of view rotations it is a reliable geometric cue to 3-D shape and viewer/object motion. The authors summarize why structure from motion algorithms are often very sensitive to errors in the measured image velocities and then show how to efficiently and reliably extract an incomplete qualitative solution. They also show how to augment this into a complete solution if additional constraints or views are available. A real-time example is presented in which the 3-D visual interpretation of hand gestures or a hand-held object is used as part of a man-machine interface. This is an alternative to the Polhemus coil instrumented Dataglove commonly used in sensing manual gestures.>

The authors present a navigation system using multiple sensors for unknown and dynamic indoor environments. To achieve robustness and flexibility in the mobile robot, a behavior-based system architecture is developed consisting of multilayered behaviors using multiple sensors that were ultrasonic sensors and a video camera. Basic behaviors required for navigation, such as avoiding obstacles, moving toward free space, and following targets are redundantly developed as agents and combined in a behavior-based system architecture. The capabilities of the system were demonstrated in unstructured real office environments using an indoor mobile robot.>

The authors are developing a compact cart-type robot system which moves around an office. They propose a behavior-based architecture with three clustered (reflexive, purposive, and adaptive) agents that realizes efficiency in attaining the mission of the robot, and robustness against the various kinds of failures that may occur in a dynamic environment. The reflexive-level group consists of agents with contact, infrared, and ultrasonic sensors which maintain minimal safety of the robot. The role of the purposive-level group is to achieve the global mission of the robot, such as 'if the robot detects a small fire in the office, find and reach a fire extinguisher as soon as possible'. The adaptive-level group stands by to recover from failure in the purposive-level group or in a deadlock situation. Experimental results showed the effectiveness of the method.>

We present a navigation system using multiple sensors for unknown and dynamic indoor environments. To achieve the robustness and flexibility of the mobile robot, we propose a new behavior-based architecture with three groups of clustered (reflexive, purposive, and adaptive) agents that realizes both efficiency in attaining the mission of the robot and robustness against the various kinds or failures that may occur in a dynamic environment. Basic behaviors required for navigation, such as, avoiding obstacles, moving towards free space, and following targets, are redundantly developed as agents and combined in the behavior-based system architecture.

Reports a system for detecting human like moving objects in time-varying images. The authors show how it is possible to detect the image trajectories of people moving in ordinary indoor scenes. The system consists of three subprocesses: changing region detection, moving object tracking and movement interpretation. The processes are executed in parallel so hat each one can recover from the others' errors. This ensures the reliable detection of the trajectories in difficult cases such as movement across complicated backgrounds. The authors have built a trial detection system using a parallel image processing system. The details of the trial system and experimental results of walking person detection are described.>

A robot vision system that automatically generates an object recognition strategy from a 3D model and recognizes the object using this strategy is presented. The appearance of an object from various viewpoints is described in terms of visible 2D features such as parallel lines and ellipses. Features are then ranked according to the number of viewpoints from which they are visible. The rank and feature extraction cost of each feature are used to generate a treelike strategy graph. This graph gives an efficient feature search order when the viewpoint is unknown, starting with commonly occurring features and ending with features specific to a certain viewpoint. The system searches for features in the order indicated by the graph. After detection, the system compares a lines representation generated from the 3D model with the image features to localize the object. Perspective projection is used in the localization process to obtain the precise position and attitude of the object, whereas orthographic projection is used in the strategy generation process to allow symbolic manipulation. Experimental results are given.>

A vision system is presented which automatically generates an object recognition strategy from a 3D model, and recognizes the object using this strategy. In this system, the appearances of an object from various viewpoints are described with visible 2D features, such as parallel lines and ellipses. Then, the features in the appearances are ranked according to the number of viewpoints from which they are visible. The rank and the feature extraction cost for each feature are considered to generate a tree-like strategy graph. It shows an efficient feature search order when the viewpoint is unknown, starting with commonly occurring features and ending with features specific to a certain viewpoint. The system searches for features in the order indicated by the graph. After detection, the system compares the line representation generated from the 3D model and the image features to localize the object.>

A visual navigation system for a robot working in a nuclear power plant is presented. Such a system should have two functions. One is to provide a rough environment map or directions towards a goal, such as turning left at the second corner. The other is to avoid obstacles and to find a passage for continuous movement of the robot. The operation of the system is divided into two stages. In the first stage, a multi-information local map (MIL map) describing information necessary for self-location measurement is created interactively from stereo images collected during remote-controlled navigation. The second stage is autonomous navigation. The robot moves autonomously by using the map with the aid of the second function of the system. The system consists of five subsystems: an environment teaching subsystem, to create the MIL map; a self-location measurement subsystem, to find the robot's own location; an obstacle detection subsystem; a path planning subsystem; and a path following subsystem. Experimental results from a mobile robot show the usefulness of the system.>

This paper describes 3D vision techniques for an advanced robot system in terms of efficient hardware and practical software. Arm-controlled stereo cameras have been developed for efficient 3D sensing, whose interocular distance, yaw and tilt are under program control. Stereo vision programs are implemented with the aid of the stereo cameras and a high-speed vision processor. Knowledge-guided vision algorithms are realized in a high-level processor, where a priori knowledge regarding the environments, objects and procedure controls are utilized.