Human skeleton reconstruction enables smart monitoring and biomechanical analysis, yet RGB/RGB-D systems face privacy concerns in public spaces, illumination sensitivity, and occlusion. To overcome these limitations, we propose a privacy-preserving approach for 3D skeleton reconstruction using ankle-level 2D LiDAR-based Motion History Images (MHIs). Our method integrates a direction-guided multi-head deep learning model, leveraging ResNet18 as the backbone to extract features from the MHIs and classify walking directions (front, left, right, back), thereby improving pose estimation across diverse orientations. Experiments comparing ResNet18, ResNet50, and DenseNet121 demonstrate significant improvements in accuracy, with DenseNet121 showing the best performance in terms of joint position error, while ResNet18 offers the best trade-off between accuracy and inference speed for real-time deployment. An ablation study reveals the importance of direction classification, with the removal of this feature leading to increased errors, especially in joints such as the spine base. We further validate anatomical consistency by estimating subject-specific height from predicted skeletons, observing consistent accuracy across individuals. These results position our approach as a robust and privacy-conscious alternative to conventional methods for 3D skeleton reconstruction.

When an automatic wheelchair or a self-carrying robot moves along with human agents, prediction for the next possible actions by the participating agents, play an important role in realization of successful cooperation among them. In this paper, we mounted a robot to a wheelchair body so that it provides embodied projective signals to the human agents, indicating the next possible action to be performed by the wheelchair. We have analyzed how human participants, particularly when they are in a multiparty interaction, would respond to such a system in experiments. We designed two settings for the robot's projective behavior. The first design allows the robot to face towards the human agents (Face-to-Face model), and the other allows it to face forward as the human agents do, then turn around to the human agents when it indicates where the wheelchair will move to (Body-Torque model). The analysis examined reactions by the human agents to the wheelchair, his/her accompanying human agent, and others who pass by them in the experiment's setting. The results show that the Body-Torque model seems more effective in enhancing cooperative behavior among the human participants than the Face-to-Face model when they are moving to a forward direction together.

In this paper, we aim to enhance the interaction between the performer and the audience in live idol performances. We propose a system for converting the movements of individual members of an idol group into vibrations and their voices into light on handheld devices for the audience. Specifically, for each performer, the system acquires data on movement and voice magnitudes via an acceleration sensor attached to the right wrist and microphone. The obtained data is then converted into motor vibrations and lights from an LED. The receiving devices for the audience members come in the form of a pen light or doll. A prototype system was made to collect acceleration data and voice magnitude data measurements for our experiments with an idol group in Japan to verify whether the performer's movements and singing voice could be correctly measured during real live performance conditions. We developed a program to present the strength of the movements and singing voice corresponding to one of the members as vibrations and lights based on the information of the recorded data. Then, an experiment was conducted for eight subjects that observed the performance. We found that seven out of eight subjects could identify the idol performer with corresponding vibrations and lighting from the device.

In this paper, we propose a remote monitoring and communication system for elderly care. We aim to detect emergencies concerning elderly people living alone and send alerts to his/her family from a remote location. For this reason, the system has two modules: the monitoring module and the video chat module. When the monitoring module of the system detects any unusual events concerning the elderly, the system sends an alert message to their family's smart phone. Then, their family can monitor the current situation via the video chat module. The elderly and his/her family member can also easily use the video chat function for day-to-day communication without requiring any complex actions for the users. For the video chat display at the elderly side, we use a television set, which all elderly would be expected to have in their rooms. This gives our system a familiar appearance so the elderly can use it without feeling aversion towards the system. The monitoring system for detecting emergencies is a doll-like robot. Besides monitoring, the robot can also initiate simple conversations autonomously in normal situations, which can improve the elderly person's sense of familiarity towards the system.

Our memory is first acquired as memory of a personal event (episodic memory). Then, its temporal context is removed and the resultant memory is used in higher-order information processing, such as thinking, reasoning and so on. Therefore, episodic memory is the basis of our memories. In this paper, we propose an episodic memory model which adopts the physiological findings of the hippocampus. Episodic memory can be regarded as a temporal sequence of patterns. In general, such a sequence may have some patterns which appear several times in it, or plural temporal sequences may share some same patterns with each other. That is, it is indispensable for the episodic memory model to handle one-to-many association and context. In the proposed model, the neurogenesis in dentate gyrus (DG) contributes to build different representations in CA3 for the same input patterns, and facilitates one-to-many association. Moreover, the asymmetric spike timing dependent synaptic plasticity introduced to the learning of recurrent collateral in CA3 enables memory of context. Computer simulation results show that the proposed model can deal with complex temporal sequences.

In this research, we aim to reveal the difference between robot mediated multi-party human interaction and human mediated multi-party human interaction. We do this by examining and comparing the multi-party participants' interactions towards the robot human mediators in regard to the mediator's questions. From this experiment, we found that the participants engaged in interaction among each other more when the questioner was the robot. This paper suggests a concept and a design of robots, which enhances social interaction among humans because of its intervention.

In this article, we discuss our findings from an ethnographic study at an elderly care center where we observed the utilization of two different functions of human gaze to convey service order (i.e., “who is served first and who is served next”). In one case, when an elderly person requested assistance, the gaze of the care worker communicated that he/she would serve that client next in turn. In the other case, the gaze conveyed a request to the service seeker to wait until the care worker finished attending the current client. Each gaze function depended on the care worker's current engagement and other behaviors. We sought to integrate these findings into the development of a robot that might function more effectively in multiple human-robot party settings. We focused on the multiple functions of gaze and bodily actions, implementing those functions into our robot. We conducted three experiments to gauge a combination of gestures and gazes performed by our robot. This article demonstrates that the employment of gaze is an important consideration when developing robots that can interact effectively in multiple human-robot party settings.

In this paper we describe a prototype remote communication support system to be used as a communication catalyst for dementia care. The system aims to delay the progression of dementia and helps caregivers by facilitating the social interaction of dementia patients. In the system framework, a video telecommunication system was installed in order to enable communication between patients and caregivers. Next, a system control board was designed to maintain robust operation so that users could connect with each other anytime. In the system, we also implemented a reminiscence based therapy application to prevent the progression of dementia. The application can show old photos to dementia patients during a typical video chat. From the caregivers' point of view, our system is also an important form of assistance for repeating announcements and instructions to dementia patients with poor memory. Accordingly, we also made a video replay application which can be set to replay video at a fixed time everyday. Finally, we made aesthetically pleasing robots that serve to make interaction with the system more natural for dementia patients.

We propose a human-robot interaction approach for social robots that attracts and controls the attention of a target person depending on her/his current visual focus of attention. The system detects the person's current task (attention) and estimates the level by using the “task-related contextual cues” and “gaze pattern.” The attention level is used to determine the suitable time to attract the target person's attention toward the robot. The robot detects the interest or willingness of the target person to interact with it. Then, depending on the level of interest of the target person, the robot generates awareness and establishes a communication channel with her/him. To evaluate the performance, we conducted an experiment using our static robot to attract the target human's attention when she/he is involved in four different tasks: reading, writing, browsing, and viewing paintings. The proposed robot determines the level of attention of the current task and considers the situation of the target person. Questionnaire measures confirmed that the proposed robot outperforms a simple attention control robot in attracting participants' attention in an acceptable way. It also causes less disturbance and establishes effective eye contact. We implemented the system into a commercial robotic platform (Robovie-R3) to initiate interaction between visitors and the robot in a museum scenario. The robot determined the visitors' gaze points and established a successful interaction with a success rate of 91.7%.

In this paper, we propose a human-robot interaction system in which the robot detects and classifies the target human's gaze pattern into either spontaneous looking or scene-relevant looking. If the gaze pattern is detected as the spontaneous looking, the robot waits for the target human without disturbing his/her attention. However, if the gaze pattern is detected as the scene-relevant looking, the robot establishes a communication channel with him/her in order to explain about the scene. We have implemented the proposed system into a robot, Robovie-R3 as a museum guide robot and tested the system to confirm its effectiveness.

In order to reduce the burden of caregivers facing an increased demand for care, particularly for the elderly, we developed a system whereby multiple robotic wheelchairs can automatically move alongside a companion. This enables a small number of people to assist a substantially larger number of wheelchair users effectively. This system utilizes an environmental map and an estimation of position to accurately identify the positional relations among the caregiver (or a companion) and each wheelchair. The wheelchairs are consequently able to follow along even if the caregiver cannot be directly recognized. Moreover, the system is able to establish and maintain appropriate positional relations.

In this paper, we propose a robot system able to take visitors on guided tours in a museum. When developing a robot capable of giving a tour of an actual museum, it is necessary to implement a robot system able to measure the location and orientation of the visitors using bearing sensors installed in a specific environment. Furthermore, the robot needs information pertaining to both tour attendees and other visitors in the vicinity, in order to effectively lead a tour for the attendees. Therefore, we propose a new robot system consisting of simple elements such as a laser range finder attached to a pole. By merely placing the sensor poles, we can track the location and orientation of visitors. We employ a group detection method to distinguish tour attendees from other people around the robot. In contrast to previous group detection methods our method can recognize a group of people even when they are not walking. In experiments our system successfully tracked and grouped visitors in the vicinity even when they were just standing.

We are developing a helper robot able to fetch objects requested by users. This robot tries to recognize objects through verbal interaction with the user concerning the objects that it cannot detect autonomously. We have shown that the system can recognize objects based on an ontology for interaction. In this paper, we extend a human description ontology to link a "human description" to "attributes of objects" for our interactive object recognition framework. We develop an interactive object recognition system based on this ontology. Experimental results confirmed that the system could efficiently recognize objects by utilizing this ontology.

We have been conducting research on ways to contribute to reducing the workload of caregivers. We developed a wheelchair capable of following a single accompanying caregiver in a parallel position. However, observations in actual care facilities revealed that people often move as a group: for instance, two caregivers each pushed a wheelchair together. Therefore, we now aim to develop a robotic wheelchair system that allows multiple wheelchairs and accompanying caregivers to coordinate their movement together. However, such a situation necessitates a complex system. Making such a system practical represents an enormous challenge. Therefore, as a first step we decided to break down this complex “multiple-multiple” situation by isolating particular instances with multiple wheelchairs and/or multiple caregivers. In this paper, we focus on the “single-multiple” situation; i.e., a situation where a single wheelchair user is accompanied by multiple accompanies. We propose a robotic wheelchair system that facilitates coordinated movement between the wheelchair and the caregivers. In other words, the wheelchair system will track multiple people in the vicinity using GPGPU, and distinguish the accompanying caregivers from among other passersby based on the trajectory of the individuals' movement.

In this paper, we propose a robot system which can take visitors on guided tours in a museum. When we consider developing a robot capable of giving a tour of an actual museum, we must implement a robot system able to measure the location and orientation of the robot and visitors using bearing sensors installed in a specific environment. Although many previous methods employed markers or tags attached to the robot to obtain positional information through cameras, it is not easy to situate sensors in the environment itself. SLAM is also used to localize the position of the robot. However, its robustness may not be sufficient when the system is deployed in an actual real-life situation since there will be many people moving freely around the robot. On the other hand, the robot needs information pertaining to both tour attendees and other visitors in the vicinity, in order to give a tour for the attendees. Therefore, we propose a new robot system which consists of simple devices such as multiple laser range finders attached to a pole. By just placing the sensor poles, we could track the location and orientation of the robot and visitors at the same time. We then conducted experiments to confirm the effectiveness and accuracy of our system. In addition, we conducted demonstrative experiments that our robot takes three visitors to the guide tour.

We are developing a service robot that provides assisted-care, such as serving tea to the elderly in care facilities. From the sociological analysis of human-human interaction in care facilities, we have found that people often use nonverbal behaviors such as raising their hand to make a request to caregivers. In this paper, we propose a request gesture recognition method for our service robot. In day care facilities, many people are moving around. Hence it is a difficult problem to correctly recognize request gestures among various movements. We have solved this problem by exploiting the sociological interaction analysis result. We have found that a person who would like to make a request to a caregiver waits for the moment when the caregiver looks in the direction towards the person, and then shows a request signal such as raising his/her hand. This means that the robot needs to analyze human motions only when it finds a frontal face in the direction of its face. We have developed a robot system to perform such request recognition and confirmed its effectiveness through experiments.

Recent research in HRI has emphasized the need to design affective interaction systems equipped with social intelligence. A robot's awareness of its social role encompasses the ability to behave in a socially acceptable manner, the ability to communicate appropriately according to the situation, and the ability to detect the feelings of interactive partners, as humans do with one another. In this paper, we propose an intelligent robotic method of attracting a target person's attention in a way congruent to satisfying these social requirements. If the robot needs to initiate communication urgently, such as in the case of reporting an emergency, it does not need to consider the current situation of the person it is addressing. Otherwise, the robot should observe the person to ascertain who or what s/he is looking at (VFOA), and how attentively s/he is doing so (VFOA level). Moreover, the robot must identify an appropriate time at which to attract the target person's attention so as to not interfere with his/her work. We have realized just such a robotic system by developing computer vision methods to detect a target person's VFOA and its level, and testing the system's effectiveness in a series of experiments.

Although it is desirable for wheelchair users to move alone by operating wheelchairs themselves, they are often pushed or accompanied by caregivers, especially in elderly care facilities. In designing robotic wheelchairs, therefore, it is important to consider how to reduce the caregiver's load and support their activities. We particularly focus on communication among wheelchair users and caregivers because face-to-face communication is known to be effective in ameliorating elderly mental health issues. Hence, we propose a maneuverable robotic wheelchair that can move autonomously towards a goal while considering the situation of the accompanying caregiver. Our robotic wheelchair starts to move autonomously towards the goal as soon as the caregiver sets the goal position, such as a bathroom, by utilizing its touch panel interface. While moving towards the goal, the robotic wheelchair autonomously controls its speed and direction of movement to facilitate natural communication between the wheelchair user and the caregiver, by observing the situation of the accompanying caregiver.

Museum guide robots should observe visitors in order to identify those who may desire a guide, and then initiate conversation with them. This paper presents a model for such robot behavior. Initiation of conversation is an important concern for social service robots such as museum guide robots. When people enter into a social interaction, they tend to situate themselves in a spatial-orientational arrangement such that each is facing inward around a space to which each has immediate access. When this kind of particular spatial formation occurs, they can feel that they are participating in the conversation; once they perceive their participation, they will subsequently try to maintain this spatial formation. We developed a model that describes the constraints and expected behaviors in initiation of conversation. Experimental results demonstrate that our model significantly improves a robot's performance in initiating conversation.

A major challenge is to design a robot that can attract and control human attention in various social situations. If a robot would like to communicate a person, it may turn its gaze to him/her for eye contact. However, it is not an easy task for the robot to make eye contact because such a turning action alone may not be enough in all situations, especially when the robot and the human are not facing each other. In this paper, we present an attention control approach through robot's behaviors that can attract a person's attention by three actions: head turning, head shaking, and uttering reference terms corresponding to three viewing situations in which the human vision senses the robot (near peripheral field of view, far peripheral field of view, and out of field of view). After gaining attention, the robot makes eye contact through showing gaze awareness by blinking its eyes, and directs the human attention by eye and head turning behaviors to share an object.

From our ethnographic studies on various kinds of museums, we discovered that guides routinely propose questions to visitors in order to draw their attention towards both his/her explanation and the exhibit. The guides' question sequences tend to begin with a pre-question which serves to not only monitor visitors' behavior and responses, but to also alert visitors that a primary question would follow. We implemented this questioning-strategy with our robot system and investigated whether this strategy would also work in human-robot interaction. We developed a vision system that enables the robot to choose an appropriate visitor by monitoring a visitor's response from the initiation of a pre-question to the following pause. Results indicate that this questioning-strategy works effectively in human-robot interaction. In this experiment, the robot asked visitors about a photograph. At the pre-question, the robot delivered a rather easy question followed by a more challenging question (Figure 1). More participants turned their head away from the exhibition when they were not sure about their answer to the question. They either faced away from the robot, or smiled wryly at the robot or at each other. These types of behaviors index participants' states of knowledge, which we could utilize to develop a system by which the robot could choose an appropriate candidate by computational recognition.

In this paper, we present our work designing a robot that explains an exhibit to multiple visitors in a museum setting, based on ethnographic analysis of interactions between expert human guides and visitors. During the ethnographic analysis, we discovered that expert human guides employ some identical strategies and practices in their explanations. In particular, one of these is to involve all visitors by posing a question to an appropriate visitor among them, which we call the "creating a puzzle" sequence. This is done in order to draw visitors' attention towards not only the exhibit and but also the guide's explanation. While creating a puzzle, the human guide can monitor visitors' responses and choose an "appropriate" visitor (i.e. one who is likely to provide an answer). Based on these findings, sociologists and engineers together developed a guide robot that coordinates verbal and non-verbal actions in posing a question or "a puzzle" that will draw visitors' attention, and then explain the exhibit for multiple visitors. During the explanation, the robot chooses an "appropriate" visitor. We tested the robot at an actual museum. The results show that our robot increases visitors' engagement and interaction with the guide, as well as interaction and engagement among visitors.

A mobile museum guide robot is expected to establish a proper spatial formation with the visitors. After observing the videotaped scenes of human guide-visitors interaction at actual museum galleries, we have developed a mobile robot that can guide multiple visitors inside the gallery from one exhibit to another. The mobile guide robot is capable of establishing spatial formation known as "F-formation" at the beginning of explanation. It can also use a systematic procedure known as "pause and restart" depending on the situation through which a framework of mutual orientation between the speaker (robot) and visitors is achieved. The effectiveness of our method has been confirmed through experiments.

Recently, several robotic/intelligent wheelchairs equipped with user-friendly interfaces and/or autonomous functions to fulfill set goals have been proposed. Although wheelchair users may wish to go out alone, caregivers often accompany them. Therefore, it is important to consider how to reduce the caregivers' load while supporting their activities and facilitating communication between the caregiver and wheelchair user. Hence, we have proposed a robotic wheelchair that can move alongside a caregiver by observing his/her behavior, such as body position and orientation. This form of motion enables easy communication between the wheelchair user and the caregiver. However, in our previous system, the caregiver had to set their own initial position by tapping the touchscreen displaying the distance data image captured by a laser range sensor. In this paper, therefore, we introduce a new interface we designed by incorporating an omni-directional camera so that the accompanying caregiver can easily set the initial position by tapping his/her facial image instead. Moreover, when the system loses the accompanying caregiver due to them making an unexpected motion, it promptly re-identifies the caregiver by using color appearance cues. These functions improve the usefulness of our robotic wheelchair and heighten the possibility of our wheelchair being utilized in our daily life.

A major challenge in HRI is to design a social robot that can attract a target human's attention to control his/her attention toward a particular direction in various social situations. If a robot would like to initiate an interaction with a person, it may turn its gaze to him/her for eye contact. However, it is not an easy task for the robot to make eye contact because such a turning action alone may not be enough to initiate an interaction in all situations, especially when the robot and the human are not facing each other or the human intensely attends to his/her task. In this paper, we propose a conceptual model of attention control with four phases: attention attraction, eye contact, attention avoidance, and attention shift. In order to initiate an attention control process, the robot first tries to gain the target participant's attention toward it through head turning, or head shaking action depending on the three viewing situations where the robot is captured in his/her field of view (central field of view, near peripheral field of view, and far peripheral field of view). After gaining her/his attention, the robot makes eye contact only with the target person through showing gaze awareness by blinking its eyes, and directs her/his attention toward an object by turning its eyes and head cues. Moreover, the robot can show attention to aversion behaviors if non-target persons look at it. We design a robot based on the proposed approach, and it is confirmed as effective to control the target participant's attention in experimental evaluation.

A social robot needs to attract the attention of a target human and shift it from his/her current focus to what is sought by the robot. The robot should recognize the current target's attention level to smoothly perform this attention control. In this paper, we propose a vision-based system to detect the level of attention or willingness of the target person towards the robot and to control his/her attention. The system estimates the attention level from rich visual cues of human's face and head. Then, by timing target's attention to determine the appropriate attention level, it generates aware signals and makes eye contact with the target. Finally, the robot shifts the target's attention to an intended direction. The experimental results reveal that the proposed system is effective in controlling the target's attention.

It is a major challenge in HRI to design a social robot that is able to direct a target human's attention towards an intended direction. For this purpose, the robot may first turn its gaze to him/her in order to establish eye contact. However, such a turning action of the robot may not in itself be sufficient to make eye contact with the target person in all situations, especially when the robot and the person are not facing each other or the human is intensely engaged in a task. In this paper, we propose a conceptual model of attention control with five phases: attention attraction, eye contact, attention avoidance, gaze back, and attention shift. We conducted two experiments to validate our model in human-robot interaction scenarios.

This paper presents our ongoing work developing service robots that provide assisted-care, such as serving tea to the elderly in care facilities. In multi-party settings, a robot is required to be able to deal with requests from multiple individuals simultaneously. In particular, when the service robot is concentrating on taking care of a specific person, other people who want to initiate interaction may feel frustrated with the robot. To a considerable extent this may be caused by the robot's behavior, which does not indicate any response to subsequent requests while preoccupied with the first. Therefore, we developed a robot that can display acknowledgement, in a socially acceptable manner, to each person who wants to initiate interaction. In this paper we focus on the task of tea-serving, and introduce a robot able to bring tea to multiple users while accepting multiple requests. The robot can detect a person's request (raising their hand) and move around people using its localization system. When the robot detects a person's request while serving tea to another person, it displays its acknowledgement by indicating "Please wait" through a nonverbal action. Because it can indicate its acknowledgement of their requests socially, people will likely feel more satisfied with the robot even when it cannot immediately address their needs.

This video presents our ongoing work developing a robotic wheelchair that can move automatically alongside a caregiver. Recently, several robotic/intelligent wheelchairs possessing autonomous functions for reaching a goal and/or user-friendly interfaces have been proposed. Although ideally wheelchair users may wish to go out alone, they are often accompanied by caregivers. Therefore, it is important to consider how to reduce the caregivers' load and support their activities and facilitate communication between the wheelchair user and caregiver. Moreover, a sociologist pointed out that when a wheelchair user is accompanied by a companion, the latter is inevitably seen as a caregiver [1]. In other words, the equality of the relationship is publicly undermined when the wheelchair is pushed by a companion. Hence, we propose a robotic wheelchair which can move alongside a caregiver or companion, and facilitate easy communication between them and the wheelchair user. However, it is not always desirable for a caregiver to be alongside a wheelchair. For instance, a caregiver may step in front of the wheelchair to open a door, and pedestrians may be encumbered by the wheelchair and companion if they move along side-by-side in a narrow corridor. To cope with these problems, our robotic wheelchair can move alongside a caregiver collaboratively depending on the circumstances. A laser range sensor is employed to track the caregiver and observe the environment around the wheelchair [2]. When obstacles are detected in the wheelchair's path of motion, it adjusts its position accordingly. In the video we demonstrate these functions of our robotic wheelchair. We are now conducting experiments to confirm the effectiveness of our wheelchair at an elderly care center in Japan.

This paper introduces a service robot that provides assisted-care, such as serving tea to the elderly in care facilities. In multi-party settings, a robot is required to be able to deal with requests from multiple individuals simultaneously. In particular, when the service robot is concentrating on taking care of a specific person, other people who want to initiate interaction may feel frustrated with the robot. To a considerable extent this may be caused by the robot's behavior, which does not indicate any response to subsequent requests while preoccupied with the first. Therefore, we developed a robot that can project the order of service in a socially acceptable manner to each person who wishes to initiate interaction. In this paper we focus on the task of tea-serving, and introduce a robot able to bring tea to multiple users while accepting multiple requests. The robot can detect a person raising their hand to make a request, and move around people using its mobile functions while avoiding obstacles. When the robot detects a person's request while already serving tea to another person, it projects that it has received the order by indicating “you are next” through a nonverbal action, such as turning its gaze to the person. Because it can project the order of service and indicate its acknowledgement of their requests socially, people will likely feel more satisfied with the robot even when it cannot immediately address their needs. We confirmed the effectiveness of this capability through an experiment in which the robot distributed snacks to participants.

This paper presents a method of selecting the answerer from audiences for a museum guide robot. We performed the preliminary experiments that a robot distributed its gaze towards visitors to select an answerer and analyzed visitors' responses. From these experiments, we have found that the visitors who are asked questions by the robot feel embarrassed when they have no prior knowledge about the question and the visitor's gaze during the question plays an important role to avoid being asked question. Based on these findings we developed functions for a guide robot to select the answerer by observing behaviors of multiple visitors. Multiple visitors' head motions are tracked and recognized by using an omni-directional camera and a laser range sensor. The robot detects the visitors' positive and negative responses by observing their head motions while asking questions. We confirmed the effectiveness of our method by experiments.

Object category detection with large appearance variation is a fundamental problem in computer vision. The appearance of object categories can change due to intra-class variability, viewpoint, and illumination. For object categories with large appearance change a sub-categorization based approach is necessary. This paper proposes a sub-category optimization approach that automatically divides an object category into an appropriate number of sub-categories based on appearance variation. Instead of using a predefined intra-category sub-categorization based on domain knowledge or validation datasets, we divide the sample space by unsupervised clustering based on discriminative image features. Then the clustering performance is verified using a sub-category discriminant analysis. Based on the clustering performance of the unsupervised approach and sub-category discriminant analysis results we determine an optimal number of sub-categories per object category. Extensive experimental results are shown using two standard and the authors' own databases. The comparison results show that our approach outperforms the state-of-the-art methods.

In this paper, we report on our development of a robotic system that assists people in accomplishing simple tasks in daily life (e.g., retrieving objects for handicapped and elderly people). These tasks, inevitably involve detecting various kinds of objects. In particular, here, we present an interactive method to detect objects using spatial information. Our experimental results confirm the usefulness and efficiency of our system. We also show how the approach can be improved and highlight necessary directions for future research.

This paper presents a method in a museum guide robot to choose an appropriate answerer among multiple visitors. First, we observed and videotaped scenes of gallery talk when human guides ask visitors questions. Based on an analysis of this video, we have found that the guides selects an answerer by distributing his or her gaze towards multiple visitors and observing the visitors' gaze responses during the question. Then, we performed experiments on a robot that distributes its gaze towards multiple visitors, and analyzed visitors' responses. From these experiments, we found that visitors who are asked questions by the robot felt embarrassed when he or she had no prior knowledge about the questions, and that visitor gaze during the questions plays an important role in avoiding being asked questions. Based on these findings, we have developed a function in a guide robot that observes visitor gaze response and selects an appropriate answerer based on these responses. Gaze responses are tracked and recognized using an omnidirectional camera and a laser range sensor. The effectiveness of our method was confirmed through experiments.

Designing technologies that support the explanation of museum exhibits is a challenging domain. In this paper we develop an innovative approach - providing a robot guide with resources to engage visitors in an interaction about an art exhibit. We draw upon ethnographical fieldwork in an art museum, focusing on how tour guides interrelate talk and visual conduct, specifically how they ask questions of different kinds to engage and involve visitors in lengthy explanations of an exhibit. From this analysis we have developed a robot guide that can coordinate its utterances and body movement to monitor the responses of visitors to these. Detailed analysis of the interaction between the robot and visitors in an art museum suggests that such simple devices derived from the study of human interaction might be useful in engaging visitors in explanations of complex artifacts.

In this paper, we propose a new example of non-holonomic mobile robot, which we call the surface walker. This robot is composed of a hemisphere-shaped shell and a 2-d.o.f. mass-control device (pendulum) inside it, and undergoes the rolling contact constraint between the hemispherical surface of the robot and the ground. Unlike a lot of non-holonomic robots which have ever been researched, the drift term exists in the system of the hemisphere robot. First, we show basic concepts which the hemisphere robot has, and construct the kinetic model of this robot. Then we realize the locomotion control of the robot by periodically oscillating the internal pendulum and show its effectiveness by control experiments.

Recently, several robotic/intelligent wheelchairs have been proposed that employ user-friendly interfaces or autonomous functions. Although it is often desirable for user to operate wheelchairs on their own, they are often accompanied by a caregiver or companion. In designing wheelchairs, it is important to reduce the caregiver load. In this paper we propose a robotic wheelchair that can move with a caregiver side by side. In contrast to a front-behind position, in a side-by-side position it is more difficult for wheelchairs to adjust when the caregiver makes a turn. To cope with this problem we present a visual-laser tracking technique. In this technique, a laser range sensor and an omni-directional camera are integrated to observe the caregiver. A Rao-Blackwellized particle filter framework is employed to track the caregiver's position and orientation of both body and head based on the distance data and panorama images captured from the laser range sensor and the omni-directional camera. After presenting this technique, we introduce an application of the wheelchair for museum visit use.

Nonverbal behavior plays an important role in human communication. In this paper, we propose a novel museum guide robot that has three different types of communication modes (autonomous, remote-control, and remote-interactive), which are integrated to interact with visitors through nonverbal behavior. First, the autonomous mode is an autonomously controlled mode in which the robot can directly interact with visitors through nonverbal behaviors such as head movements or arm gestures. Second, when a visitor begins to ask the robot questions, the communication is changed into a remote-control mode. In this mode, a remote operator controls the robot with gestures to interact with visitors. Third, in the remote-interactive mode, a remote operator can also interact with the robot with voice and gestures through a network. In particular, the remote operator can ask the robot to do something with nonverbal behaviors such as pointing while saying, “Bring this brochure to him.” These three communication modes are switched seamlessly depending on the situation.

Service robots need to be able to recognize and identify objects located within complex backgrounds. Since no single method may work in every situation, several methods need to be combined. However, there are several cases when autonomous recognition methods fail. We propose an interactive recognition method in these cases. To develop a natural Human Robot Interaction (HRI), it is necessary that the robot should unambiguously perceive the description of an object given by human. This paper reports on our experiment in which we examined the expressions humans use in describing ordinary objects. The results show that humans typically describe objects using one of multiple colors. The color is usually either that of the object background or that of the largest object portion. Based on these results, we describe our development of a robot vision system that can recognize objects when a user adopts simple expressions to describe the objects. This research suggests the importance of connecting ‘symbolic expressions’ with the ‘real world’ in human-robot interaction.

In this paper, we propose a novel method to learn motion patterns and detect anomalies by human trajectory analysis. Human trajectories are various, for example, moving, roaming, pausing, and so on. But, current approaches for the analysis of motion patterns are effective only in understanding simple trajectories. We aim to understand complicated human trajectories with long-term observation. To deal with spatial and temporal features of trajectories, we employ HMM (Hidden Markov Model) to model time-series features of human positions. Next, a similarity matrix of HMM mutual distances is formed. MDS (Multi-Dimensional Scaling) based on eigenvector decomposition provides projected coordinates of trajectories in low-dimensional space. Then we apply k-means clustering to projected data in order to acquire human motion patterns. Anomalies can be detected by the use of likelihood scores for HMM representing motion patterns. We tested the proposed method by real-world trajectories data observed in a small store. Experimental result shows that our method accurately finds typical motion patterns and unusual trajectories.

We propose a method for real-time people tracking using multiple cameras. The particle filter framework is known to be effective for tracking people, but most of existing methods adopt only simple perceptual cues such as color histogram or contour similarity for hypothesis evaluation. To improve the robustness and accuracy of tracking more sophisticated hypothesis evaluation is indispensable. We therefore present a novel technique for human head tracking using cascaded classifiers based on AdaBoost and Haar-like features for hypothesis evaluation. In addition, we use multiple classifiers, each of which is trained respectively to detect one direction of a human head. During real-time tracking the most suitable classifier is adaptively selected by considering each hypothesis and known camera position. Our experimental results demonstrate the effectiveness and robustness of our method. 1 1

This article describes a design and implementation of an augmented desk system, named EnhancedDesk, which smoothly integrates paper and digital information on a desk. The system provides users an intelligent environment that automatically retrieves and displays digital information corresponding to the real objects (e.g., books) on the desk by using computer vision. The system also provides users direct manipulation of digital information by using the users' own hands and fingers for more natural and more intuitive interaction. Based on the experiments with our first prototype system, some critical issues on augmented desk systems were identified when trying to pursue rapid and fine recognition of hands and fingers. To overcome these issues, we developed a novel method for realtime finger tracking on an augmented desk system by introducing a infrared camera, pattern matching with normalized correlation, and a pan-tilt camera. We then show an interface prototype on EnhancedDesk. It is an application to a computer-supported learning environment, named Interactive Textbook. The system shows how effective the integration of paper and digital information is and how natural and intuitive direct manipulation of digital information with users' hands and fingers is.

This paper describes two interface prototypes which we have developed on our augmented desk interface system, EnhancedDesk. The first application is Interactive Textbook, which is aimed at providing an effective learning environment. When a student opens a page which describes experiments or simulations, Interactive Textbook automatically retrieves digital contents from its database and projects them onto the desk. Interactive Textbook also allows the student hands-on ability to interact with the digital contents. The second application is the Interactive Venn Diagram, which is aimed at supporting effective information retrieval. Instead of keywords, the system uses real objects such as books or CDs as keys for retrieval. The system projects a circle around each book; data corresponding the book are then retrieved and projected inside the circle. By moving two or more circles so that the circles intersect each other, the user can compose a Venn diagram interactively on the desk. We also describe the new technologies introduced in EnhancedDesk which enable us to implement these applications.

We introduce a fast and robust method for tracking positions of the centers and the fingertips of both right and left hands. Our method makes use of infrared camera images for reliable detection of a user's hands, and uses a template matching strategy for finding fingertips. This method is an essential part of our augmented desk interface in which a user can, with natural hand gestures, simultaneously manipulate both physical objects and electronically projected objects on a desk, e.g., a textbook and related WWW pages. Previous tracking methods which are typically based on color segmentation or background subtraction simply do not perform well in this type of application because an observed color of human skin and image backgrounds may change significantly due to protection of various objects onto a desk. In contrast, our proposed method was shown to be effective even in such a challenging situation through demonstration in our augmented desk interface. This paper describes the details of our tracking method as well as typical applications in our augmented desk interface.